Adds global metadata configuration (fixes #36)
# This file specifies how the global attributes from the input files
# are transferred to the global attributes in the output file. The configuration
# can be specified for both input and output, where the configuration for
# input is applied after loading the input data files, whereas, the output
# configuration is applied in the post-processing step of the cube. The structure of
# the I/O configurations is different:
#
# Input:
# ------
# The input configuration has three sections: default, drop, and transfer.
#
# default:
# --------
#
# default: <option>
#
# The following two options are available:
# - equalize: keep all attributes that are equal between the cubes
# - drop: remove all attributes except the ones specified under transfer
#
# By default, Climix will use equalize to remove the attributes that are not equal
# between all cubes. The default configuration can be used to remove redundant
# attributes without specifying all of their names.
#
#
# drop:
# -----
#
# drop:
# - existing_attribute
#
# The drop configuration can be used to remove specific existing attributes.
#
#
# transfer:
# ---------
#
# transfer:
# attr_name_1:
# - existing_attribute
# attr_name_2:
# - existing_attribute_1
# - existing_attribute_2
#
# The variables configuration allows the user to specify which attributes to keep
# and join. Here, `attr_name` determines the names for the output attribute, while
# `existing_attribute` determines which attributes should be stored under
# `attr_name`. If `existing_attribute` has different values between cubes, the
# *unique* values will be joined with a comma separator, e.g.,
#
# transfer:
# creation-dates:
# - creation_date
#
# creates the output attribute:
# creation-dates = "2018-06-21-T20:03:01Z, 2018-06-21-T19:59:31Z"
#
# However, if the user defines multiple existing attributes they will first be joined
# with an underscore and then the unique values will be joined with a comma separator,
# e.g.,
#
# transfer:
# tracking-and-creation:
# - tracking_id
# - creation_date
#
# creates the output attribute:
# tracking-and-creation = "1a76c102-ed05-4d84-bfeb-e4dc1c48a5dc_2018-06-21-T20:03:01Z, df8c2f0a-4126-4c05-aafa-db532304cdcc_2018-06-21-T19:59:31Z"
#
#
# Output:
# -------
# The output configuration has one section: create.
#
# create:
# ---------
#
# create:
# attr_name: "string with information to store"
#
# Here, the user can define attribute values that will be set for `attr_name`, e.g.,
#
# create:
# institution: "SMHI"
#
# creates the output attribute:
# institution = "SMHI"
#
# If the user chooses the same `attr_name` as an already existing attribute, either in
# the input configuration or in the cube's global attributes, it will be overwritten.
# It is also possible to fill in useful information that is created during the run,
# e.g., existing global attributes, the software version being used, and time.
# Valid values: {NOW}, {CLIMIX_VERSION}, {existing_attribute}
#
# transfer:
# creation_date: "{NOW}"
# software: "{CLIMIX_VERSION}"
# rcm-gcm: "rcm: {rcm} gcm: {gcm}"
#
# can create the output attributes:
# creation-date = "2023-03-31T14:52:32 UTC"
# software = "CLIMIX version 0.18.0"
# rcm-gcm: "rcm: SMHI-RCA4_v1 gcm: NCC-NorESM1-M"default config:
climix:
global_attributes:
input:
default: equalize
drop:
# CORDEX and CMIP attributes that are transferred (see below)
- driving_experiment_name
- model_id
- rcm_version_id
- driving_model_id
- tracking_id
- creation_date
- driving_model_ensemble_member
- history
- history_of_appended_files
- institution
- institute_id
- references
- product
- contact
- project_id
- experiment
- experiment_id
- driving_experiment
- initialization_method
- physics_version
- realization
- source
- table_id
- title
transfer:
scenario:
- driving_experiment_name
rcm:
- model_id
- rcm_version_id
gcm:
- driving_model_id
gcm_ensemble_member:
- driving_model_ensemble_member
tracking-id_creation-date:
- tracking_id
- creation_date
history-attribute:
- history
- history_of_appended_files
input_frequency:
- frequency
input_institution:
- institution
input_institute_id:
- institute_id
input_references:
- references
input_product:
- product
input_contact:
- contact
input_project_id:
- project_id
CORDEX_domain:
- CORDEX_domain
output:
create:
product: "climate indicator"
institution: "Swedish Meteorological and Hydrological Institute, Rossby Centre"
institute_id: SMHI
references: "https://www.smhi.se/en/research/research-departments/climate-research-at-the-rossby-centre"
creation_date: "{NOW}"
software: "{CLIMIX_VERSION}"output example with default config file (equalise):
climix -e -l info -v -o /nobackup/rossby27/users/sm_carni/data/tmp/output_files/out_tmp_tn_glob_ex1.nc -x tn /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20060101-20101231.nc /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20110101-20151231.nc --activate-config
global attributes in input files:
// global attributes:
:institution = "Helmholtz-Zentrum Geesthacht, Climate Service Center, Max Planck Institute for Meteorology" ;
:institute_id = "MPI-CSC" ;
:experiment_id = "rcp85" ;
:source = "MPI-CSC-REMO2009" ;
:model_id = "MPI-CSC-REMO2009" ;
:contact = "gerics-cordex@hzg.de" ;
:comment = "CORDEX Europe RCM REMO 0.11 deg EUR-11" ;
:references = "http://www.remo-rcm.de/" ;
:initialization_method = 1 ;
:physics_version = 1 ;
:tracking_id = "436ab648-e2de-4696-9b11-fe1ac68bb87b" ;
:CORDEX_domain = "EUR-11" ;
:driving_experiment = "MPI-M-MPI-ESM-LR, rcp85, r2i1p1" ;
:driving_model_id = "MPI-M-MPI-ESM-LR" ;
:driving_model_ensemble_member = "r2i1p1" ;
:driving_experiment_name = "rcp85" ;
:rcm_version_id = "v1" ;
:product = "output" ;
:experiment = "RCP8.5" ;
:frequency = "day" ;
:creation_date = "2016-05-10T19:55:24Z" ;
:history = "2016-05-10T19:55:24Z CMOR rewrote data to comply with CF standards and CORDEX requirements." ;
:Conventions = "CF-1.4" ;
:project_id = "CORDEX" ;
:table_id = "Table day (March 2015) 6f55fe4ad23cded422652f83a747ce32" ;
:title = "MPI-CSC-REMO2009 model output prepared for CORDEX RCP8.5" ;
:modeling_realm = "atmos" ;
:realization = 2 ;
:cmor_version = "2.9.1" ;
}
// global attributes:
:institution = "Helmholtz-Zentrum Geesthacht, Climate Service Center, Max Planck Institute for Meteorology" ;
:institute_id = "MPI-CSC" ;
:experiment_id = "rcp85" ;
:source = "MPI-CSC-REMO2009" ;
:model_id = "MPI-CSC-REMO2009" ;
:contact = "gerics-cordex@hzg.de" ;
:comment = "CORDEX Europe RCM REMO 0.11 deg EUR-11" ;
:references = "http://www.remo-rcm.de/" ;
:initialization_method = 1 ;
:physics_version = 1 ;
:tracking_id = "13096055-aa41-4c04-bab3-a86e45600ab3" ;
:CORDEX_domain = "EUR-11" ;
:driving_experiment = "MPI-M-MPI-ESM-LR, rcp85, r2i1p1" ;
:driving_model_id = "MPI-M-MPI-ESM-LR" ;
:driving_model_ensemble_member = "r2i1p1" ;
:driving_experiment_name = "rcp85" ;
:rcm_version_id = "v1" ;
:product = "output" ;
:experiment = "RCP8.5" ;
:frequency = "day" ;
:creation_date = "2016-02-04T17:37:42Z" ;
:history = "2016-02-04T17:37:42Z CMOR rewrote data to comply with CF standards and CORDEX requirements." ;
:Conventions = "CF-1.4" ;
:project_id = "CORDEX" ;
:table_id = "Table day (March 2015) 6f55fe4ad23cded422652f83a747ce32" ;
:title = "MPI-CSC-REMO2009 model output prepared for CORDEX RCP8.5" ;
:modeling_realm = "atmos" ;
:realization = 2 ;
:cmor_version = "2.9.1" ;global attributes in output file:
// global attributes:
:CORDEX_domain = "EUR-11" ;
:associated_files = "gridspecFile: gridspec_atmos_fx_MPI-CSC-REMO2009_rcp85_r0i0p0.nc" ;
:cmor_version = "2.9.1" ;
:comment = "daily-minimum near-surface (usually, 2 meter) air temperature." ;
:creation_date = "2023-04-12T09:04:03 UTC" ;
:frequency = "mon" ;
:gcm = "MPI-M-MPI-ESM-LR" ;
:gcm_ensemble_member = "r2i1p1" ;
:history-attribute = "2016-02-04T17:37:42Z altered by CMOR: Treated scalar dimension: \'height\'., 2016-05-10T19:55:23Z altered by CMOR: Treated scalar dimension: \'height\'." ;
:input_contact = "gerics-cordex@hzg.de" ;
:input_frequency = "day" ;
:input_institute_id = "MPI-CSC" ;
:input_institution = "Helmholtz-Zentrum Geesthacht, Climate Service Center, Max Planck Institute for Meteorology" ;
:input_product = "output" ;
:input_project_id = "CORDEX" ;
:input_references = "http://www.remo-rcm.de/" ;
:institute_id = "SMHI" ;
:institution = "Swedish Meteorological and Hydrological Institute, Rossby Centre" ;
:modeling_realm = "atmos" ;
:product = "climate indicator" ;
:rcm = "MPI-CSC-REMO2009_v1" ;
:references = "https://www.smhi.se/en/research/research-departments/climate-research-at-the-rossby-centre" ;
:scenario = "rcp85" ;
:software = "Climix version 0.17.0+28.g004e89e" ;
:tracking-id_creation-date = "436ab648-e2de-4696-9b11-fe1ac68bb87b_2016-05-10T19:55:24Z, 13096055-aa41-4c04-bab3-a86e45600ab3_2016-02-04T17:37:42Z" ;
:Conventions = "CF-1.7" ;
}Example with importing a config file using "-f" that has the setting,
default: dropand "-l debug"
climix -e -l debug -v -o /nobackup/rossby27/users/sm_carni/data/tmp/output_files/out_tmp_tn_glob_ex2.nc -x tn -f /home/sm_carni/Project/YML/climix_config.yml /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20060101-20101231.nc /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20110101-20151231.nc --activate-config
// global attributes:
:CORDEX_domain = "EUR-11" ;
:creation_date = "2023-04-12T09:05:14 UTC" ;
:frequency = "mon" ;
:gcm = "MPI-M-MPI-ESM-LR" ;
:gcm_ensemble_member = "r2i1p1" ;
:history-attribute = "2016-02-04T17:37:42Z altered by CMOR: Treated scalar dimension: \'height\'., 2016-05-10T19:55:23Z altered by CMOR: Treated scalar dimension: \'height\'." ;
:input_contact = "gerics-cordex@hzg.de" ;
:input_frequency = "day" ;
:input_institute_id = "MPI-CSC" ;
:input_institution = "Helmholtz-Zentrum Geesthacht, Climate Service Center, Max Planck Institute for Meteorology" ;
:input_product = "output" ;
:input_project_id = "CORDEX" ;
:input_references = "http://www.remo-rcm.de/" ;
:institute_id = "SMHI" ;
:institution = "Swedish Meteorological and Hydrological Institute, Rossby Centre" ;
:product = "climate indicator" ;
:rcm = "MPI-CSC-REMO2009_v1" ;
:references = "https://www.smhi.se/en/research/research-departments/climate-research-at-the-rossby-centre" ;
:scenario = "rcp85" ;
:software = "Climix version 0.17.0+28.g004e89e" ;
:tracking-id_creation-date = "436ab648-e2de-4696-9b11-fe1ac68bb87b_2016-05-10T19:55:24Z, 13096055-aa41-4c04-bab3-a86e45600ab3_2016-02-04T17:37:42Z" ;
:Conventions = "CF-1.7" ;
}Running a index with two different data inputs from two different models (default config file):
climix -e -o /nobackup/rossby27/users/sm_carni/data/tmp/output_files/out_tmp_dtr_ex3.nc -x dtr /home/rossby/data_lib/esgf/cordex/output/EUR-11/SMHI/NCC-NorESM1-M/rcp85/r1i1p1/RCA4/v1/day/tasmax/latest/tasmax_EUR-11_NCC-NorESM1-M_rcp85_r1i1p1_SMHI-RCA4_v1_day_20060101-20101231.nc /home/rossby/data_lib/esgf/cordex/output/EUR-11/SMHI/NCC-NorESM1-M/rcp85/r1i1p1/RCA4/v1/day/tasmax/latest/tasmax_EUR-11_NCC-NorESM1-M_rcp85_r1i1p1_SMHI-RCA4_v1_day_20110101-20151231.nc /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20060101-20101231.nc /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20110101-20151231.nc -l debug --activate-config
produces the output attributes:
// global attributes:
:CORDEX_domain = "EUR-11" ;
:creation_date = "2023-04-12T09:07:17 UTC" ;
:frequency = "mon" ;
:gcm = "NCC-NorESM1-M, MPI-M-MPI-ESM-LR" ;
:gcm_ensemble_member = "r2i1p1, r1i1p1" ;
:history-attribute = "2016-05-10T19:55:23Z altered by CMOR: Treated scalar dimension: \'height\'., 2016-02-04T17:37:42Z altered by CMOR: Treated scalar dimension: \'height\'." ;
:input_contact = "gerics-cordex@hzg.de, rossby.cordex@smhi.se" ;
:input_frequency = "day" ;
:input_institute_id = "SMHI, MPI-CSC" ;
:input_institution = "Swedish Meteorological and Hydrological Institute, Rossby Centre, Helmholtz-Zentrum Geesthacht, Climate Service Center, Max Planck Institute for Meteorology" ;
:input_product = "output" ;
:input_project_id = "CORDEX" ;
:input_references = "http://www.smhi.se/en/Research/Research-departments/climate-research-rossby-centre, http://www.remo-rcm.de/" ;
:institute_id = "SMHI" ;
:institution = "Swedish Meteorological and Hydrological Institute, Rossby Centre" ;
:product = "climate indicator" ;
:rcm = "MPI-CSC-REMO2009_v1, SMHI-RCA4_v1" ;
:references = "https://www.smhi.se/en/research/research-departments/climate-research-at-the-rossby-centre" ;
:scenario = "rcp85" ;
:software = "Climix version 0.17.0+28.g004e89e" ;
:tracking-id_creation-date = "df8c2f0a-4126-4c05-aafa-db532304cdcc_2018-06-21-T19:59:31Z, 436ab648-e2de-4696-9b11-fe1ac68bb87b_2016-05-10T19:55:24Z, acfc60b7-57a5-4b02-b03b-3e556c48efe8_2018-06-21-T19:56:04Z, 13096055-aa41-4c04-bab3-a86e45600ab3_2016-02-04T17:37:42Z" ;
:Conventions = "CF-1.7" ;
}There are some debug logging for the removed and replaced attributes, e.g.,
INFO:root:Applying global attribute input configuration
DEBUG:root:Following attributes were not found in all cubes: ['history', 'history_of_appended_files']
DEBUG:root:Attributes were removed to equalize cubes: <['model_id : MPI-CSC-REMO2009', 'title : MPI-CSC-REMO2009 model output prepared for CORDEX RCP8.5', 'realization : 2', 'associated_files = gridspecFile: gridspec_atmos_fx_MPI-CSC-REMO2009_rcp85_r0i0p0.nc', 'rossby_grib_path = /home/rossby/prod/201801/raw/', 'contact : rossby.cordex@smhi.se', 'physics_version : 1', 'institution : Helmholtz-Zentrum Geesthacht, Climate Service Center, Max Planck Institute for Meteorology', 'driving_experiment : NCC-NorESM1-M, rcp85, r1i1p1', 'creation_date : 2016-05-10T19:55:24Z', 'rossby_run_id = 201801', 'cmor_version = 2.9.1', 'institute_id : SMHI', 'model_id : SMHI-RCA4', 'driving_model_ensemble_member : r2i1p1', 'references : http://www.remo-rcm.de/', 'rossby_comment = 201801: CORDEX Europe 0.11 deg | RCA4 v1 | NCC-NorESM1-M | r1i1p1 | rcp85 | L40 | PRINCIPLES', "history : 2016-05-10T19:55:23Z altered by CMOR: Treated scalar dimension: 'height'.", 'experiment_id : rcp85', 'institute_id : MPI-CSC', 'table_id : Table day (March 2015) 6f55fe4ad23cded422652f83a747ce32', 'tracking_id : 13096055-aa41-4c04-bab3-a86e45600ab3', 'comment = daily-minimum near-surface (usually, 2 meter) air temperature.', 'creation_date : 2018-06-21-T19:59:31Z', 'driving_model_ensemble_member : r1i1p1', 'experiment : RCP8.5', 'modeling_realm = atmos', 'source : MPI-CSC-REMO2009', 'institution : Swedish Meteorological and Hydrological Institute, Rossby Centre', 'tracking_id : df8c2f0a-4126-4c05-aafa-db532304cdcc', 'contact : gerics-cordex@hzg.de', 'driving_model_id : NCC-NorESM1-M', 'creation_date : 2018-06-21-T19:56:04Z', "history : 2016-02-04T17:37:42Z altered by CMOR: Treated scalar dimension: 'height'.", 'tracking_id : acfc60b7-57a5-4b02-b03b-3e556c48efe8', 'initialization_method : 1', 'product : output', 'driving_experiment : MPI-M-MPI-ESM-LR, rcp85, r2i1p1', 'c3s_disclaimer = This data has been produced in the context of the C3S_34b_Lot1 and Lot 2 projects (PRINCIPLES/CORDEX4CDS) as a data provider for the Climate Data Store within the Copernicus Climate Change Service (C3S - https://climate.copernicus.eu/). While abiding by the highest scientific and technical standards, ECMWF cannot warrant that any information provided by the C3S will be entirely free from errors or omissions or that such errors or omissions can or will be rectified entirely. This applies to data from projects that continue to be developed, but are made publicly available for the purpose of feedback and testing. Some data or metadata may have been created or structured in files or formats that are not error-free. ECMWF accepts no responsibility with regard to such problems incurred as a result of using this data (see http://climate.copernicus.eu/disclaimer-privacy for the full disclaimer)', 'driving_experiment_name : rcp85', 'tracking_id : 436ab648-e2de-4696-9b11-fe1ac68bb87b', 'driving_model_id : MPI-M-MPI-ESM-LR', 'references : http://www.smhi.se/en/Research/Research-departments/climate-research-rossby-centre', 'rcm_version_id : v1', 'creation_date : 2016-02-04T17:37:42Z', 'project_id : CORDEX']>.
INFO:root:Applying global attribute output configuration
DEBUG:root:Following attributes were replaced: ['product = output'] #Note: This only shows for attributes that was not removed under drop and were replaced. In the new default file product is removed so this line won't show in the output.