What is the recommend method of generating cubes from files?

[tiagoams]

When generating cubes from files, including regridding, application of masks and creation of new variables, is xcube gen the recommended method of should xcube gen2 be used? (asking for latest release v1.2.0)

Also, will custom input processors work with gen2, or is there another way of reading files that are n ot read by the default processor? For instance when spatial dimentions are not named "lat" and "lon".

Finally, is there a way to aggregate or interpolate in time when creating a datacube? For instance to create a monthly average cube from daily files.

Thanks!

[TonioF]

Hello @tiagoams,

I suggest to use xcube gen2. For xcube gen2 documentation is provided here: https://xcube.readthedocs.io/en/latest/generator.html . There is also a section on details of specifying inputs.

There are plans of adding temporal resampling, but so far it is not included.

[tiagoams]

Thanks @TonioF,

I am trying to migrate from gen to gen2, using the CLI and am trying to make sense of the new parameters.
I can generate a cube from a single file, but how do I specify multiple files in data_id, say "*.nc"?

(...)
  "input_config": {
    "store_id": "file",
    "store_params": {
        "root": "F:\\Data\\OCEANCOLOUR_ATL_BGC_L4_MY_009_114\\cmems_obs-oc_atl_bgc-plankton_my_l4-multi-1km_P1M\\1998"
    },
    "data_id": "*.nc"
  },
(...)

returns

ERROR: Data resource "*.nc" does not exist in store

[TonioF]

Also note that we have notebooks explaining the use of xcube gen2: https://github.com/dcs4cop/xcube/tree/master/examples/notebooks/generators

[TonioF]

Thanks @TonioF,

I am trying to migrate from gen to gen2, using the CLI and am trying to make sense of the new parameters. I can generate a cube from a single file, but how do I specify multiple files in data_id, say "*.nc"?

(...)
  "input_config": {
    "store_id": "file",
    "store_params": {
        "root": "F:\\Data\\OCEANCOLOUR_ATL_BGC_L4_MY_009_114\\cmems_obs-oc_atl_bgc-plankton_my_l4-multi-1km_P1M\\1998"
    },
    "data_id": "*.nc"
  },
(...)

returns

ERROR: Data resource "*.nc" does not exist in store

You cannot use wildcards as a data_id. If you have multiple inputs, you need to specify multiple input_configs, which you can do like this:

(...)
  "input_configs": [
	{
	    "store_id": "file",
            "store_params": {
                "root": "F:\\Data\\OCEANCOLOUR_ATL_BGC_L4_MY_009_114\\cmems_obs-oc_atl_bgc-plankton_my_l4-multi-1km_P1M\\1998"
            },
            "data_id": "1.nc"
        },
	{
	    "store_id": "file",
            "store_params": {
                "root": "F:\\Data\\OCEANCOLOUR_ATL_BGC_L4_MY_009_114\\cmems_obs-oc_atl_bgc-plankton_my_l4-multi-1km_P1M\\1998"
	    },
            "data_id": "2.nc"
        },
	(...)
  ],
(...)

To prevent boilerplate code (and multiple store creations), it is advisable to pass an additional store config file which could look like this:

local_nc_files:
    store_id: file
    store_params: 
        root: F:\\Data\\OCEANCOLOUR_ATL_BGC_L4_MY_009_114\\cmems_obs-oc_atl_bgc-plankton_my_l4-multi-1km_P1M\\1998

The input_configs section would then become this:

(...)
  "input_configs": [
	{
	    "store_id": "@local_nc_files",
            "data_id": "1.nc"
	},
	{
	    "store_id": "@local_nc_files",
            "data_id": "2.nc"
	},
	(...)
  ],
(...)