climix merge requestshttps://git.smhi.se/climix/climix/-/merge_requests2021-02-22T10:30:34Zhttps://git.smhi.se/climix/climix/-/merge_requests/145Fix double _freq_ elements in output files (closes #214)2021-02-22T10:30:34ZLars BärringFix double _freq_ elements in output files (closes #214)0.13.2Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/merge_requests/71Equalise cubes (towards #36)2021-02-22T11:35:01ZLars BärringEqualise cubes (towards #36)Updated copy of `iris.experimental.equalise_cubes` that now returns the deleted attributes of each cube in the list.
Maybe we should remove part/all of the Crown copyright ?Updated copy of `iris.experimental.equalise_cubes` that now returns the deleted attributes of each cube in the list.
Maybe we should remove part/all of the Crown copyright ?Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/merge_requests/72Global metadata (closes #36)2021-02-22T11:36:25ZLars BärringGlobal metadata (closes #36)Process global attributes according to "instructions" in a default yaml file located in the climix/climix/util directory and possibly overridden by "instructions" in a user supplied yaml file. Includes hook for tracking_id.
Ready for r...Process global attributes according to "instructions" in a default yaml file located in the climix/climix/util directory and possibly overridden by "instructions" in a user supplied yaml file. Includes hook for tracking_id.
Ready for review.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/merge_requests/154Remove empty/inon-existing directories from input file list (closes #230)2021-07-06T15:10:53ZLars BärringRemove empty/inon-existing directories from input file list (closes #230)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/merge_requests/73WIP: Index function beginning of spell2022-09-15T00:56:49ZLars BärringWIP: Index function beginning of spellBeginning of spell with delay. "use case" is the *beginning of growing season* part of ETCCDI **`gsl`**Beginning of spell with delay. "use case" is the *beginning of growing season* part of ETCCDI **`gsl`**https://git.smhi.se/climix/climix/-/merge_requests/155Updated guess_output_template with code from midas, after some changes (close...2022-09-28T13:32:20ZLars BärringUpdated guess_output_template with code from midas, after some changes (closes #231)0.16Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/merge_requests/151Dtr reducer as parameter instead of hard-coded (closes #187)2022-09-28T13:42:09ZLars BärringDtr reducer as parameter instead of hard-coded (closes #187)Picking up this again as #208 has been fixed. Please review critically as I am here extrapolating my understanding maybe a bit too far.Picking up this again as #208 has been fixed. Please review critically as I am here extrapolating my understanding maybe a bit too far.0.15Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/merge_requests/181Update Fork with changes in Upstream2022-10-19T13:29:19ZCarolina NilssonUpdate Fork with changes in UpstreamCarolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/merge_requests/186fix for different tas cell_methods2022-11-17T09:11:02ZJoakim Löwfix for different tas cell_methodsfix for different tas cell_methodsfix for different tas cell_methodsKlaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/merge_requests/146Restructure logging to become more user-oriented (closes #190)2022-11-17T09:55:35ZLars BärringRestructure logging to become more user-oriented (closes #190)Note the comment in [main.py line 120](https://git.smhi.se/a001257/climix/-/commit/7967c3023a1a1ce91050dae96d7ca9636dd2b388#note_12701) regarding possible improvement (that I do not know how to solve (via a class??)Note the comment in [main.py line 120](https://git.smhi.se/a001257/climix/-/commit/7967c3023a1a1ce91050dae96d7ca9636dd2b388#note_12701) regarding possible improvement (that I do not know how to solve (via a class??)0.16Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/merge_requests/194ThresholdedRunningStatistics.lazy_func(...) should call RunningStatistics.laz...2022-12-05T10:09:54ZJoakim LöwThresholdedRunningStatistics.lazy_func(...) should call RunningStatistics.lazy_func(...) (fixes #265)`ThresholdedRunningStatistics.lazy_func(...)` called `RunningStatistics.call_func(...)`, changed to call `RunningStatistics.lazy_func(...)` instead. Have tested with `rx5day` and `rx{ND}day` (using `rx1day` and `rx3day`) with same result...`ThresholdedRunningStatistics.lazy_func(...)` called `RunningStatistics.call_func(...)`, changed to call `RunningStatistics.lazy_func(...)` instead. Have tested with `rx5day` and `rx{ND}day` (using `rx1day` and `rx3day`) with same results before and after change.0.16https://git.smhi.se/climix/climix/-/merge_requests/173Added thresholded DTR index function2023-01-30T14:35:49ZRenate WilckeAdded thresholded DTR index functionAdded thresholded DTR index function. Function calculates number of days with DTR above flexible threshold. Closes issue #195.Added thresholded DTR index function. Function calculates number of days with DTR above flexible threshold. Closes issue #195.https://git.smhi.se/climix/climix/-/merge_requests/179Draft: Index function for diurnal temperature range with threshold (fixes #195)2023-01-30T14:36:10ZJoakim LöwDraft: Index function for diurnal temperature range with threshold (fixes #195)Replaces !173.
Added a `ThresholdedDiurnalTemperatureRange` index function and `DifferenceThresholdMixin` mixin class to deal with differences (in temperature).
When converting a temperature difference from e.g. Celsius to Kelvin, or C...Replaces !173.
Added a `ThresholdedDiurnalTemperatureRange` index function and `DifferenceThresholdMixin` mixin class to deal with differences (in temperature).
When converting a temperature difference from e.g. Celsius to Kelvin, or Celsius to Fahrenheit, it's not enough to use `convert_units` on the difference value (as noted in #195). The idea in `DifferenceThresholdMixin` is to convert both the difference value and the zero of the original unit, and then create the difference in the new unit using both converted values. E.g. difference in Celsius of 10 = (10 - 0) gives a difference in Kelvin of (283.15 - 273.15) = 10, or a difference in Fahrenheit of (50 - 32) = 18. I think this should work as long as there is a linear relationship between the two units.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/merge_requests/206Draft: Pre commit test2023-02-13T13:38:56ZJoakim LöwDraft: Pre commit testThis is only for CI/CD testingThis is only for CI/CD testinghttps://git.smhi.se/climix/climix/-/merge_requests/180Draft:Test index_functions.support module2023-02-16T08:14:37ZJoakim LöwDraft:Test index_functions.support moduleI've added test functions for the index_functions/support.py module. Some comments:
- I'm using helper classes inheriting the mixin classes to test them (the mixin classes).
- Added CubeHelper and CoordHelper classes wrapped in fixtures...I've added test functions for the index_functions/support.py module. Some comments:
- I'm using helper classes inheriting the mixin classes to test them (the mixin classes).
- Added CubeHelper and CoordHelper classes wrapped in fixtures in conftest.py for helping with producing testdata (cubes and coords), I'm planning to use them for the other modules in index_functions/* as well.
I've started adding tests for index_functions/index_functions.py but want some feedback before I get ahead of myself. Do the test approach seem ok this far? An alternative would be to work with test-classes instead of test-functions, the iris project seems to do that. Do you have any preference either way?
Also, I'll add an issue for this merge before un-drafting it.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/merge_requests/198Draft: Reference period defined in Index_definition.yml and command line arg ...2023-02-24T10:45:47ZCarolina NilssonDraft: Reference period defined in Index_definition.yml and command line arg -reference_period (fixes #273)The reference period is read from the index_definition.yml file as a parameter with 'kind' time_period, further information in #273. Can be specified from command line with e.g., `-reference_period "['1971','2019']"`. The reference perio...The reference period is read from the index_definition.yml file as a parameter with 'kind' time_period, further information in #273. Can be specified from command line with e.g., `-reference_period "['1971','2019']"`. The reference period is added as a global attribute `reference_period: "1971 to 2019"`.
TODO:
- Update time format (see comments)
- See if there is a possibility to store the reference period using !200 and then remove self.extra_global_attr.
- Add some kind of check wheather the ref. period is outside of data timerange.0.17 (Black Lion)https://git.smhi.se/climix/climix/-/merge_requests/204Index function for requested indices (fixes #282 #283)2023-03-01T10:42:14ZCarolina NilssonIndex function for requested indices (fixes #282 #283)Index function to compute thresholded occurrences. Name: CountThresholdedOccurrences, suggestion for another name?
Use -f to import metadata file with index definitions:
```
indices:
txgt0rrge1:
reference: SMHI
default_period:...Index function to compute thresholded occurrences. Name: CountThresholdedOccurrences, suggestion for another name?
Use -f to import metadata file with index definitions:
```
indices:
txgt0rrge1:
reference: SMHI
default_period: annual
output:
var_name: "txgt0rrge1"
standard_name:
proposed_standard_name:
long_name: "Number of precipitation days with daily minimum temperature above zero degC (Precip >= 1 and Tmin > 0)"
units: "1"
cell_methods:
- time: sum within days
- time: sum over days
input:
low_data: tasmin
high_data: pr
index_function:
name: count_thresholded_occurrences
parameters:
threshold_low_data:
kind: quantity
standard_name: air_temperature
long_name: "Level crossing value for daily air temperature"
data: 0
units: "degree_Celsius"
threshold_high_data:
kind: quantity
standard_name: lwe_precipitation_rate
long_name: "Wet day threshold"
data: 1
units: "mm day-1"
logical_condition:
kind: logical
base_operator:
logical:
and:
- low_data > threshold_low_data
- high_data >= threshold_high_data
ET:
short_name:
long_name:
definition:
comment:
txgt13lt18:
reference: SMHI
default_period: annual
output:
var_name: "txgt13lt18"
standard_name:
proposed_standard_name:
long_name: "Number of days with daily maximum temperature above thirteen degC and below eighteen degC (13 < Tmax < 18)"
units: "1"
cell_methods:
- time: sum within days
- time: sum over days
input:
data: tasmax
index_function:
name: count_thresholded_occurrences
parameters:
threshold_low_data:
kind: quantity
standard_name: air_temperature
long_name: "Level crossing value for daily air temperature"
data: 13
units: "degree_Celsius"
threshold_high_data:
kind: quantity
standard_name: air_temperature
long_name: "Level crossing value for daily air temperature"
data: 18
units: "degree_Celsius"
logical_condition:
kind: logical
base_operator:
logical:
and:
- low_data > threshold_low_data
- high_data < threshold_high_data
ET:
short_name:
long_name:
definition:
comment:
```0.17 (Black Lion)https://git.smhi.se/climix/climix/-/merge_requests/200Adding a attribute_setup.yml metadata file for global metadata attributes2023-03-10T14:26:17ZCarolina NilssonAdding a attribute_setup.yml metadata file for global metadata attributesA new metadata file containing the attribute setups has been added to "climix/climix/etc". This file can be replaced with the "-f" flag if the user wants to add their own settings.
Attribute file description:
```
# This file specifies t...A new metadata file containing the attribute setups has been added to "climix/climix/etc". This file can be replaced with the "-f" flag if the user wants to add their own settings.
Attribute file description:
```
# This file specifies the how the 'global attributes' metadata from the input files are
# transferred to the 'global attributes' in the output file. This includes actions for
# creating new, removing, renaming, replacing, appending values, keeping, and joining,
# global attributes. The 'order' determines the order of which the actions are performed.
# E.g., if the 'tracking_id' attribute is droped before it is joined the value will not
# exist and cannot not be joined. The settings can be specified for both the 'input
# attributes' and 'output attributes'. The settings for the 'input attributes' are applied
# after loading the input datafiles and the settings for the 'output attributes' are
# applied in the post_processing step of the cube. It is important that the settings
# are applied at the correct step in the process. E.g., attributes that may be removed
# in the concatenation process needs to be transferred in the 'input attributes' to be
# stored in the output file. However, replacing attributes in the 'input attributes'
# can change the look of the history from the input files and replacing values should
# be done in the 'output attributes' to ensure that the history is intact.
#
# order:
# - action_name_1
# - action_name_2
# Description: The order of which the actions are performed. The action at the top
# will be performed first, etc.
# join:
# - output_attribute_name_1 : attribute_name_1
# - output_attribute_name_1 : attribute_name_2
# - output_attribute_name_2 : attribute_name_3
# Description: Joins together the global attributes, given by 'attribute_name' for
# each input file, that has the same 'output_attribute_name'. Each
# 'output_attribute_name' are stored in the output files global attributes.
# The input attribute is removed when it is joined.
# append:
# - output_attribute_name : attribute_value
# Description: Appends a value to an already existing global attribute with the same
# output_attribute_name. Creates a new global attribute if the name does not
# exist and stores it as 'output_attribute_name' in the output global
# attributes.
# rename:
# - output_attribute_name : attribute_name
# Description: Takes a global attribute given by 'attribute_name' and stores it as
# 'output_attribute_name' in the output files global attributes. The
# input attribute is removed when it is renamed.
# replace:
# - attribute_name: new_value
# Description: Replaces the global attribute value for the attribute given by 'attribute_name'
# with 'new_value'.
# keep:
# - attribute_name
# Description: Keeps the global attribute value given by 'attribute_name'. This is useful when
# using 'unspecified' for the drop action.
# drop:
# - attribute_name
# Description: Removes the global attribute given by 'attribute_name'. 'unspecified' can be used
# to remove all of the unspecified attributes, i.e. only the attributes that have
# been appended, renamed, replaced, kept, or, joined is stored. 'unspecified' can be
# used for either the 'output_attributes', 'input_attributes' or both. If it is given
# for the 'input_attributes', all unspecified values that is not used will be removed
# before concatenation. This makes it easier to see if some attributes that we want to
# keep is removed by iris. However, it will not remove values that are created in the
# post processing of the cube. If 'unspecified' is used for the 'output_attributes'
# some attributes that are created in the post processing of the cube might be removed,
# e.g. 'proposed_standard_name' and 'frequency', these can still be kept using the keep
# action.
```
default attribute setups:
```
input_attributes:
order:
- replace
- append
- keep
- join
- rename
- drop
drop:
- unspecified
join:
- senario : driving_experiment_name
- rcm : model_id
- rcm : rcm_version_id
- gcm : driving_model_id
- gcm_ensemble_member : driving_model_ensemble_member
- tracking-id_creation-date : tracking_id
- tracking-id_creation-date : creation_date
- history-attributes : history
- history-attributes : history_of_appended_files
- comments : rossby_comment
- comments : comment
- comments : driving_experiment_comment
rename:
- input_frequency : frequency
- input_institution : institution
- input_institute_id : institute_id
- input_references : references
- input_product : product
output_attributes:
order:
- append
- replace
- join
- keep
- drop
append:
- product: "climate indicator"
- institution: "Swedish Meteorological and Hydrological Institute, Rossby Centre"
- institute_id: SMHI
- references: "https://www.smhi.se/en/research/research-departments/climate-research-at-the-rossby-centre"
- creation_date: "$NOW"
- tracking_id: "$TRACKING_ID"
- software: "$CLIMIX_VERSION"
replace:
- title: "--- TODO ---"
keep:
- frequency
- CORDEX_domain
- proposed_standard_name
```
Output when running:
`climix -e -l debug -v -p annual -x fd /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20060101-20101231.nc /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20110101-20151231.nc`
Information text and warnings with debug:
```
6510ms:metadata.py:prepare_joined_attributes() DEBUG:root:Failed to join attribute <['history_of_appended_files', 'rossby_comment', 'driving_experiment_comment']> it does not exist in the cube data.
6511ms:metadata.py:prepare_joined_attributes() DEBUG:root:Failed to join attribute <['history_of_appended_files', 'rossby_comment', 'driving_experiment_comment']> it does not exist in the cube data.
6511ms:metadata.py:perform_actions() INFO:root:Performs action <join> on attributes <[{'senario': 'driving_experiment_name'}, {'rcm': 'model_id'}, {'rcm': 'rcm_version_id'}, {'gcm': 'driving_model_id'}, {'gcm_ensemble_member': 'driving_model_ensemble_member'}, {'tracking-id_creation-date': 'tracking_id'}, {'tracking-id_creation-date': 'creation_date'}, {'history-attributes': 'history'}, {'history-attributes': 'history_of_appended_files'}, {'comments': 'rossby_comment'}, {'comments': 'comment'}, {'comments': 'driving_experiment_comment'}]>
6511ms:metadata.py:perform_actions() INFO:root:Performs action <rename> on attributes <[{'input_frequency': 'frequency'}, {'input_institution': 'institution'}, {'input_institute_id': 'institute_id'}, {'input_references': 'references'}, {'input_product': 'product'}]>
6511ms:metadata.py:perform_actions() INFO:root:Performs action <drop> on attributes <['unspecified']>
6512ms:metadata.py:perform() DEBUG:root:Failed to store attributes <['institution', 'driving_model_ensemble_member', 'comment', 'software', 'rossby_comment', 'rcm_version_id', 'creation_date', 'driving_experiment_comment', 'driving_experiment_name', 'driving_model_id', 'institute_id', 'history_of_appended_files', 'product', 'tracking_id', 'frequency', 'model_id', 'references', 'history']> they does not exist in the cube data. This can occur when the values did not exist in the input file or they where renamed or joined.
6512ms:metadata.py:perform() DEBUG:root:Failed to store attributes <['institution', 'driving_model_ensemble_member', 'comment', 'software', 'rossby_comment', 'rcm_version_id', 'creation_date', 'driving_experiment_comment', 'driving_experiment_name', 'driving_model_id', 'institute_id', 'history_of_appended_files', 'product', 'tracking_id', 'frequency', 'model_id', 'references', 'history']> they does not exist in the cube data. This can occur when the values did not exist in the input file or they where renamed or joined.
6512ms:metadata.py:check_removed_attributes() INFO:root:Removed 'unspecified' attributes: ['associated_files', 'physics_version', 'table_id', 'initialization_method', 'experiment', 'experiment_id', 'Conventions', 'project_id', 'driving_experiment', 'realization', 'cmor_version', 'contact', 'modeling_realm', 'source']
8290ms:main.py:do_main() DEBUG:root:Calculating index
8291ms:index.py:__call__() DEBUG:root:Starting preprocess
8291ms:index.py:__call__() DEBUG:root:Finished preprocess
8292ms:index.py:__call__() DEBUG:root:Data found for input <data>
8293ms:index.py:__call__() DEBUG:root:Adding coord categorisation.
9133ms:index.py:__call__() DEBUG:root:Preparing cubes
9134ms:index.py:__call__() DEBUG:root:Setting up aggregation
9283ms:aggregators.py:compute_pre_result() DEBUG:root:Setting up pre-result in aggregate mode
9304ms:aggregators.py:compute_pre_result() DEBUG:root:Setup completed in 0
9307ms:metadata.py:perform_actions() INFO:root:Performs action <append> on attributes <[{'product': 'climate indicator'}, {'institution': 'Swedish Meteorological and Hydrological Institute, Rossby Centre'}, {'institute_id': 'SMHI'}, {'references': 'https://www.smhi.se/en/research/research-departments/climate-research-at-the-rossby-centre'}, {'creation_date': '$NOW'}, {'tracking_id': '$TRACKING_ID'}, {'software': '$CLIMIX_VERSION'}]>
9307ms:metadata.py:fill_in_value() INFO:root:Filled value <$NOW> with <2023-02-03T13:19:57 UTC>
9307ms:metadata.py:fill_in_value() INFO:root:Filled value <$TRACKING_ID> with <--- TODO --->
9307ms:metadata.py:fill_in_value() INFO:root:Filled value <$CLIMIX_VERSION> with <CLIMIX version 0.15.0+19.gae27e31>
9307ms:metadata.py:perform_actions() INFO:root:Performs action <replace> on attributes <[{'title': '--- TODO ---'}]>
9307ms:metadata.py:perform_actions() INFO:root:Performs action <keep> on attributes <['frequency', 'CORDEX_domain', 'proposed_standard_name']>
```
Output global attributes:
```
// global attributes:
:CORDEX_domain = "EUR-11" ;
:comments = "daily-minimum near-surface (usually, 2 meter) air temperature." ;
:creation_date = "2023-02-03T13:19:57 UTC" ;
:frequency = "yr" ;
:gcm = "MPI-M-MPI-ESM-LR" ;
:gcm_ensemble_member = "r2i1p1" ;
:history-attributes = "2016-02-04T17:37:42Z altered by CMOR: Treated scalar dimension: height., 2016-05-10T19:55:23Z altered by CMOR: Treated scalar dimension: height." ;
:input_frequency = "day" ;
:input_institute_id = "MPI-CSC" ;
:input_institution = "Helmholtz-Zentrum Geesthacht, Climate Service Center, Max Planck Institute for Meteorology" ;
:input_product = "output" ;
:input_references = "http://www.remo-rcm.de/" ;
:institute_id = "SMHI" ;
:institution = "Swedish Meteorological and Hydrological Institute, Rossby Centre" ;
:product = "climate indicator" ;
:rcm = "MPI-CSC-REMO2009_v1" ;
:references = "https://www.smhi.se/en/research/research-departments/climate-research-at-the-rossby-centre" ;
:senario = "rcp85" ;
:software = "CLIMIX version 0.15.0+19.gae27e31" ;
:title = "--- TODO ---" ;
:tracking-id_creation-date = "436ab648-e2de-4696-9b11-fe1ac68bb87b_2016-05-10T19:55:24Z, 13096055-aa41-4c04-bab3-a86e45600ab3_2016-02-04T17:37:42Z" ;
:tracking_id = "--- TODO ---" ;
:Conventions = "CF-1.7" ;
}
```
Error handling have been added such that any improper use will give a warning or an understandable error message.
Test case:
Importing a attribute setup file containing several 'errors' which triggering warning messages:
run:
`climix -e -l debug -p annual -x fd /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20060101-20101231.nc /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20110101-20151231.nc -f /home/sm_carni/Project/YML/attribute_setups_1.yml`
For attribute setups:
```
input_attributes:
drop:
- tracking_ID
- creation_date
join:
- input_joined_attributes: tracking_ID
- input_joined_attributes: creation_date
append:
- input_unit: Degree_Celsius
replace:
- product: climate service
rename:
- input_unit: Fahrenheit
- unit : input_unit
output_attributes:
order:
- append
- replace
- join
- drop
append:
- {frequency : dayss}
- input_unit : C
join:
- output_joined_attributes : input_unit
- output_joined_attributes : output_unit
replace:
- output_unit : Kelvin
```
Information and warnings:
```
WARNING:root:Replacing input attributes can change the 'look' of the history of the input datafiles (only use this if you want to replace or rename historical values). To replace values in the output file use replace for the output attributes.
DEBUG:root:Failed to join attribute <['tracking_ID']> it does not exist in the cube data.
DEBUG:root:Failed to join attribute <['tracking_ID']> it does not exist in the cube data.
INFO:root:Performs action <drop> on attributes <['tracking_ID', 'creation_date']>
DEBUG:root:Failed to drop attribute <['tracking_ID', 'creation_date']> it does not exist in the cube data.
DEBUG:root:Failed to drop attribute <['tracking_ID', 'creation_date']> it does not exist in the cube data.
INFO:root:Performs action <join> on attributes <[{'input_joined_attributes': 'tracking_ID'}, {'input_joined_attributes': 'creation_date'}]>
INFO:root:Performs action <append> on attributes <[{'input_unit': 'Degree_Celsius'}]>
INFO:root:Performs action <replace> on attributes <[{'product': 'climate service'}]>
INFO:root:Performs action <rename> on attributes <[{'input_unit': 'Fahrenheit'}, {'unit': 'input_unit'}]>
DEBUG:root:Failed to rename attribute <['Fahrenheit']> it does not exist in the cube data.
DEBUG:root:Failed to rename attribute <['Fahrenheit']> it does not exist in the cube data.
WARNING:root:Attributes removed by iris: <[{'history': "2016-05-10T19:55:23Z altered by CMOR: Treated scalar dimension: 'height'.", 'tracking_id': '436ab648-e2de-4696-9b11-fe1ac68bb87b'}, {'history': "2016-02-04T17:37:42Z altered by CMOR: Treated scalar dimension: 'height'.", 'tracking_id': '13096055-aa41-4c04-bab3-a86e45600ab3'}]>. Note: To transfer values that are incompatible between cubes use the join action in the attribute setups.
DEBUG:root:Calculating index
DEBUG:root:Starting preprocess
DEBUG:root:Finished preprocess
DEBUG:root:Data found for input <data>
DEBUG:root:Adding coord categorisation.
DEBUG:root:Preparing cubes
DEBUG:root:Setting up aggregation
DEBUG:root:Setting up pre-result in aggregate mode
DEBUG:root:Setup completed in 0
DEBUG:root:Failed to join attribute <['input_unit', 'output_unit']> it does not exist in the cube data.
INFO:root:Performs action <append> on attributes <[{'frequency': 'dayss'}, {'input_unit': 'C'}]>
INFO:root:Performs action <replace> on attributes <[{'output_unit': 'Kelvin'}]>
DEBUG:root:Failed to replace attribute <['output_unit']> it does not exist in the cube data
INFO:root:Performs action <join> on attributes <[{'output_joined_attributes': 'input_unit'}, {'output_joined_attributes': 'output_unit'}]>
```
Output:
```
// global attributes:
:CORDEX_domain = "EUR-11" ;
:associated_files = "gridspecFile: gridspec_atmos_fx_MPI-CSC-REMO2009_rcp85_r0i0p0.nc" ;
:cmor_version = "2.9.1" ;
:comment = "daily-minimum near-surface (usually, 2 meter) air temperature." ;
:contact = "gerics-cordex@hzg.de" ;
:driving_experiment = "MPI-M-MPI-ESM-LR, rcp85, r2i1p1" ;
:driving_experiment_name = "rcp85" ;
:driving_model_ensemble_member = "r2i1p1" ;
:driving_model_id = "MPI-M-MPI-ESM-LR" ;
:experiment = "RCP8.5" ;
:experiment_id = "rcp85" ;
:frequency = "yr, dayss" ;
:initialization_method = 1 ;
:input_joined_attributes = "2016-02-04T17:37:42Z, 2016-05-10T19:55:24Z, 2016-02-04T17:37:42Z, 2016-05-10T19:55:24Z" ;
:input_unit = "C" ;
:institute_id = "MPI-CSC" ;
:institution = "Helmholtz-Zentrum Geesthacht, Climate Service Center, Max Planck Institute for Meteorology" ;
:model_id = "MPI-CSC-REMO2009" ;
:modeling_realm = "atmos" ;
:physics_version = 1 ;
:product = "climate service" ;
:project_id = "CORDEX" ;
:rcm_version_id = "v1" ;
:realization = 2 ;
:references = "http://www.remo-rcm.de/" ;
:source = "MPI-CSC-REMO2009" ;
:table_id = "Table day (March 2015) 6f55fe4ad23cded422652f83a747ce32" ;
:title = "MPI-CSC-REMO2009 model output prepared for CORDEX RCP8.5" ;
:unit = "Degree_Celsius" ;
:Conventions = "CF-1.7" ;
}
```
Errors in the attribute setups such as missing global attributes only log a warning and does not end the run. If some attributes where removed by iris the user is also warned. For more problematic errors the run is interrupted and a error message is given.
If attributes that the user wants to keep differ between datafiles they are removed by iris. One option is to join these attributes, instead of rename/append/etc., then they will be kept. However, we need to decide on which these attributes are and put them under join.
TODO:
- Update the default attribute setup file.
- Test that the output stores all the necessary attributes for different files.
- Test that the output stores all the necessary attributes for different files using index with two datafiles.0.17 (Black Lion)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/merge_requests/217Draft: Parametrized integration tests using subset of NGCD dataset (fixes #275)2023-04-20T09:43:01ZJoakim LöwDraft: Parametrized integration tests using subset of NGCD dataset (fixes #275)Added integration/regression tests for climix with input from the NGCD dataset.
Dataset folders and input file patterns are specified in `tests/integration/configuration.yml`, and lists in `tests/integration/configuration.py` are used t...Added integration/regression tests for climix with input from the NGCD dataset.
Dataset folders and input file patterns are specified in `tests/integration/configuration.yml`, and lists in `tests/integration/configuration.py` are used to parameterize test functions in `tests/integration/test_indices.py`. Example usages:
```
# Run all tests
pytest
# Run only integrations tests
pytest tests/integration
# Run only climix integration tests with ngcd data
pytest tests -k "test_ngcd_reg"
# or
pytest tests/integration/test_indices.py::test_ngcd_reg
# Run only climix integration tests with ngcd data for index cdd
pytest tests -k "test_ngcd_reg and cdd"
# or
pytest tests/integration/test_indices.py::test_ngcd_reg -k "cdd"
```
Indices tested are: `cdd`, `cfd`, `csu`, `cwd`, `dtr`, `fd`, `id`, `prcptot`, `r1mm`, `r10mm`, `r20mm`, `rx1day`, `sdii`, `su`, `tnn`, `tnx`, `tr`, `txn`, `txx`https://git.smhi.se/climix/climix/-/merge_requests/220Prepare pre-release 0.18.0rc12023-04-21T15:28:15ZKlaus ZimmermannPrepare pre-release 0.18.0rc1