climix issueshttps://git.smhi.se/climix/climix/-/issues2019-05-17T11:05:30Zhttps://git.smhi.se/climix/climix/-/issues/44cwd has wrong operator2019-05-17T11:05:30ZKlaus Zimmermanncwd has wrong operatorcwd uses > instead of the correct >=. This was done due to problems with the standard name, but is not acceptable.cwd uses > instead of the correct >=. This was done due to problems with the standard name, but is not acceptable.0.4.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/45Deal with missing standard names2019-05-27T14:16:42ZKlaus ZimmermannDeal with missing standard namesFor many indices there are no or no appropriate standard names in CF now (for an example, see #44).
We have to decide how to deal with this.
Basically, we have four options, I think:
1. Choose one of the available standard names
2. De...For many indices there are no or no appropriate standard names in CF now (for an example, see #44).
We have to decide how to deal with this.
Basically, we have four options, I think:
1. Choose one of the available standard names
2. Decide on the standard names ourselves and ignore the table
3. Suggest a standard name for inclusion in the table and hold off on the index until approval
4. Leave the standard name empty and put a preliminary standard name in an extra attribute.
@a001977, @a001257, what do you think?https://git.smhi.se/climix/climix/-/issues/46Introduce the new column `proposed_standard_name` in the master metadata table2019-05-14T11:45:52ZLars BärringIntroduce the new column `proposed_standard_name` in the master metadata tablecf. #45. Ping Klaus when donecf. #45. Ping Klaus when done0.4.0Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/26Add dask setup to cli options2019-09-26T11:13:02ZKlaus ZimmermannAdd dask setup to cli optionshttps://git.smhi.se/climix/climix/-/issues/47Generate a new metadata.yaml from master metadata table2019-05-29T09:34:13ZLars BärringGenerate a new metadata.yaml from master metadata tablecf. #45. Await #46 to be closed. Ping Lars and Renate when done.cf. #45. Await #46 to be closed. Ping Lars and Renate when done.0.5.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/48Check if the new attribute `proposed_standard_name`requires anycode updates2019-05-14T15:12:35ZLars BärringCheck if the new attribute `proposed_standard_name`requires anycode updatescf. #45 (#46 and #47)cf. #45 (#46 and #47)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/49SU index2019-05-17T15:18:48ZRenate WilckeSU indexcount of days when daily maximum temperature is above plus 25 degCcount of days when daily maximum temperature is above plus 25 degC0.4.0Renate WilckeRenate Wilckehttps://git.smhi.se/climix/climix/-/issues/75Calculation of reference climatologies (ref IRIS)2020-09-29T15:13:55ZLars BärringCalculation of reference climatologies (ref IRIS)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/9HWDI2020-09-29T14:39:04ZKlaus ZimmermannHWDIHeat wave duration w.r.t. mean of reference period (TX > TXnorm)Heat wave duration w.r.t. mean of reference period (TX > TXnorm)https://git.smhi.se/climix/climix/-/issues/8Tg90p2020-09-29T14:38:39ZKlaus ZimmermannTg90pWarm days percent w.r.t. 90th percentile of reference period (TG > TGn90)Warm days percent w.r.t. 90th percentile of reference period (TG > TGn90)https://git.smhi.se/climix/climix/-/issues/54Add `count_occurrences` index function2019-05-16T13:55:23ZKlaus ZimmermannAdd `count_occurrences` index function0.4.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/55Add common operators2019-05-16T13:58:39ZKlaus ZimmermannAdd common operatorsWe would like to have the common operators <= and >= together with the common reducer sum.We would like to have the common operators <= and >= together with the common reducer sum.0.4.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/56Check role of reducer in `count_occurrences` index function2019-05-17T14:34:53ZKlaus ZimmermannCheck role of reducer in `count_occurrences` index functionAt the moment `count_occurrences` purports to take a reducer argument, but it is not clear what that would do.
Either the description should be clarified or the argument be removed.At the moment `count_occurrences` purports to take a reducer argument, but it is not clear what that would do.
Either the description should be clarified or the argument be removed.0.4.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/58Move entire calculation to index functions2019-05-29T15:31:35ZKlaus ZimmermannMove entire calculation to index functionsAt the moment, the aggregator calls an appropriate `apply_along_axis` with the index function only providing the inner part of this. This restricts the index functions unnecessarily in many cases and can lead to sub-optimal performance.At the moment, the aggregator calls an appropriate `apply_along_axis` with the index function only providing the inner part of this. This restricts the index functions unnecessarily in many cases and can lead to sub-optimal performance.0.6.0https://git.smhi.se/climix/climix/-/issues/59Formatting errors in r10mm index2019-05-17T12:50:31ZKlaus ZimmermannFormatting errors in r10mm indexThere are a few formatting errors in the definition for the index r10mm.
This prevents climix from running.There are a few formatting errors in the definition for the index r10mm.
This prevents climix from running.0.4.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/62Group indices to facilitate metadata templates2019-06-13T14:21:58ZLars BärringGroup indices to facilitate metadata templatesMany indices are very similar in structure and it is inefficient to have a full metadata.yaml entry for each. Grouping the indices according to their 'structure' will help to implement more efficient metadata. Possible groupings are:
FUN...Many indices are very similar in structure and it is inefficient to have a full metadata.yaml entry for each. Grouping the indices according to their 'structure' will help to implement more efficient metadata. Possible groupings are:
FUN_count: {variable}{condition}{threshold}
FUN_spell: similar
etc etc ...Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/63List of index functions2019-06-13T14:24:24ZKlaus ZimmermannList of index functionsThe following is a list of all functions that appear in the mastertable, along with the implementations in climix.
At the moment, of course, most implementations are missing. @a001257 could you prioritize the remaining ones? Or perhaps i...The following is a list of all functions that appear in the mastertable, along with the implementations in climix.
At the moment, of course, most implementations are missing. @a001257 could you prioritize the remaining ones? Or perhaps in some cases comment on their necessity?
mastertable|climix implementation
-----------|---------------------
FUN_count|count_occurrences
FUN_csdi|
FUN_dd|
FUN_dtr|
FUN_frac|
FUN_growing|
FUN_max|
FUN_mean|
FUN_min|
FUN_prctile|
FUN_spell|spell_length
FUN_wsdi|Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/74Allow spells to extend beyond the actual calculation period (as in CLIMPACT2)2020-09-22T08:56:18ZLars BärringAllow spells to extend beyond the actual calculation period (as in CLIMPACT2)cf. #72cf. #720.13https://git.smhi.se/climix/climix/-/issues/65General question about `REDUCERS`2019-05-31T10:08:01ZLars BärringGeneral question about `REDUCERS`In main.py the list of REDUCERS are somewhat limited and expanded as needed. Fair enough, but it is not clear to me if/how this should be handled more systematic. For example, should we use np.mean, or np.nanmean, or np.ma.mean? This que...In main.py the list of REDUCERS are somewhat limited and expanded as needed. Fair enough, but it is not clear to me if/how this should be handled more systematic. For example, should we use np.mean, or np.nanmean, or np.ma.mean? This question comes up asa consequence of the thest data I happened to use for !41Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/66Correct output units2019-05-29T10:51:49ZKlaus ZimmermannCorrect output unitsAt the moment, output units are simply assigned as the units from the metadata.
This worked, because for spell_length the units don't depend on the input. However, when in general the units need to be computed by the index_function, not ...At the moment, output units are simply assigned as the units from the metadata.
This worked, because for spell_length the units don't depend on the input. However, when in general the units need to be computed by the index_function, not the aggregator.0.5.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/67Fix output unit handling2019-05-29T12:19:52ZKlaus ZimmermannFix output unit handlingSince the changing of units usually changes the data itself, it can not be handled in the `update_metadata` routine of the aggregator, but instead must take place in its `post_process` method.Since the changing of units usually changes the data itself, it can not be handled in the `update_metadata` routine of the aggregator, but instead must take place in its `post_process` method.0.5.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/68Add thresholded_statistics index function2019-05-29T15:11:45ZKlaus ZimmermannAdd thresholded_statistics index functionThresholded statistics, ie statistics like mean, std, etc taking into account only those elements that satisfy a certain condition, cover a considerable class of indices, for example sdii.Thresholded statistics, ie statistics like mean, std, etc taking into account only those elements that satisfy a certain condition, cover a considerable class of indices, for example sdii.0.5.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/69Review structure of metadata2019-06-17T10:01:02ZLars BärringReview structure of metadataWe should review the structure of the metadata, and in the master table:
* new substantially version of the master table (all "ready" indices should hopefully be OK) (cf. #70)
* separate index definition and index function information,...We should review the structure of the metadata, and in the master table:
* new substantially version of the master table (all "ready" indices should hopefully be OK) (cf. #70)
* separate index definition and index function information, and "global metadata" in the metadata.yml (cf. #18)?
* move index function definition metadata part into the index function, thus help templating (cf. #62)?
* updated master table may help automating the index definitions (e.g. grandfathering #7 and !41)
* Representation of simple stats. and thresholded stats. in the master table (column "index_function"), and as index_functions (cf. #62, #63)https://git.smhi.se/climix/climix/-/issues/72How should spells spanning several periods be handled?2019-06-17T09:57:20ZKlaus ZimmermannHow should spells spanning several periods be handled?Comparing climpact2 with climix, the following difference emerged. If a spell spans more than one period, climpact2 treats it as a member of the last period. For example in the `cwd` index a wet spell starts in December of 1997 and ends ...Comparing climpact2 with climix, the following difference emerged. If a spell spans more than one period, climpact2 treats it as a member of the last period. For example in the `cwd` index a wet spell starts in December of 1997 and ends in January 1998, climpact2 then identifies this as the longest spell of 1998 with 34 days, whereas climix at the moment only counts the 14 days that are part of the period in question, 1998.
What behavior do we want? On the one hand it seems bad to not count particularly long spells. On the other hand that one is not part of the year and it would mean that one has to have access to at least one period before and after the time span of interest. Also, should that be counted for all periods, only the first, only the last, or the dominant one? What if a spell spans more than one full period, ie at least three periods? For annual indices that seams unlikely, but for monthly ones that is perhaps possible?Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/98How to deal with individual missing/invalid/masked timesteps?2019-09-12T11:15:50ZKlaus ZimmermannHow to deal with individual missing/invalid/masked timesteps?Ideally, the behavior with missing points in timeseries should be configurable.Ideally, the behavior with missing points in timeseries should be configurable.https://git.smhi.se/climix/climix/-/issues/105Variable name aliases2019-10-10T15:12:00ZLars BärringVariable name aliasesClimix should be able to handle well-known alternatives (aliases) for the standard CMIP variable names:
| CMIP | alternatives (case insensitive) | |
| ------ | ------ | --|
| **tas** | tasadjust, t2m, t2mean, tmean, meant, tm, tg | |
|...Climix should be able to handle well-known alternatives (aliases) for the standard CMIP variable names:
| CMIP | alternatives (case insensitive) | |
| ------ | ------ | --|
| **tas** | tasadjust, t2m, t2mean, tmean, meant, tm, tg | |
| **tasmax** | tasmaxadjust, t2max, tmax, maxt, tx | |
| **tasmin** | tasminadjust, t2min, tmin, mint, tn | |
| **pr** | pradjust, prec, precip, rr | conversion of standard name and unit might be necessary |
This initial list should be easy to extend, both with respect to new variables and new alternatives.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/106Monthly period is always climatology2019-09-26T08:09:45ZKlaus ZimmermannMonthly period is always climatologyThe `Monthly` period does coord categorization only by month. This leads to climatological calculations. The correct categorization and `output_coord` contains both year and month.The `Monthly` period does coord categorization only by month. This leads to climatological calculations. The correct categorization and `output_coord` contains both year and month.0.7.1Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/107change_pr_units only works for canonical units2019-09-26T08:09:45ZKlaus Zimmermannchange_pr_units only works for canonical unitsThe `change_pr_units` function assumes incoming data in canonical units. Also, it seems to always use `1s` as the integration time and the density of water as density when changing between amount based and rate based units, or mass based...The `change_pr_units` function assumes incoming data in canonical units. Also, it seems to always use `1s` as the integration time and the density of water as density when changing between amount based and rate based units, or mass based and thickness based units, respectively.0.7.1Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/108Logging2019-09-26T08:30:50ZKlaus ZimmermannLoggingTo follow the progress of calculation it would be useful to have better logging.
By setting up basic logging infrastructure we lay the ground works for integrating appropriate logging in future commits.To follow the progress of calculation it would be useful to have better logging.
By setting up basic logging infrastructure we lay the ground works for integrating appropriate logging in future commits.0.8.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/109Simplify formatting using f-strings2019-09-26T09:41:51ZKlaus ZimmermannSimplify formatting using f-stringsUsing f strings that are available since python 3.6, string formatting can be simplified.Using f strings that are available since python 3.6, string formatting can be simplified.0.8.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/110Improve dask setup2019-09-26T11:13:01ZKlaus ZimmermannImprove dask setupThe current way of setting up dask is too rigid. To allow for a more flexible setup we need a better mechanism.The current way of setting up dask is too rigid. To allow for a more flexible setup we need a better mechanism.0.8.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/111Make input data preparation more robust2019-10-08T11:10:58ZKlaus ZimmermannMake input data preparation more robustCurrently, input data preparation may fail at the concatenation step due to varying metadata.
We can use iris' `equalise_attributes` function to deal with this. This will in effect just remove all conflicting attributes, thus potentially...Currently, input data preparation may fail at the concatenation step due to varying metadata.
We can use iris' `equalise_attributes` function to deal with this. This will in effect just remove all conflicting attributes, thus potentially losing some metadata. Still, at the moment this is our best bet to be able to deal particularly with large datasets spanning many files.0.8.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/112Move sliced_mode to output layer2019-09-26T11:31:53ZKlaus ZimmermannMove sliced_mode to output layerSliced mode allows us to break up large dask graphs into more manageable pieces by performing the actual computation on only one period at a time. This approach might be made obsolete in the future if dask gains the ability to split grap...Sliced mode allows us to break up large dask graphs into more manageable pieces by performing the actual computation on only one period at a time. This approach might be made obsolete in the future if dask gains the ability to split graphs. In any case, this should be part of the output mechanism, but at the moment is handled at the graph construction level.
Let's move sliced mode to the output level by integrating it into a `save` method in main.0.8.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/113Store individual slices to output file2019-09-26T11:31:53ZKlaus ZimmermannStore individual slices to output fileIn sliced mode, by storing individual slices to the output file as they become available, we can avoid the need to realize the entire result in memory. Let's do this.In sliced mode, by storing individual slices to the output file as they become available, we can avoid the need to realize the entire result in memory. Let's do this.0.8.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/114Dependency versions missing2019-09-27T08:42:22ZKlaus ZimmermannDependency versions missingSome dependencies like dask, distributed, are required with a minimum version.
This should be reflected in the requirements in setup.py.Some dependencies like dask, distributed, are required with a minimum version.
This should be reflected in the requirements in setup.py.0.8.1Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/119Add option to deactivate sentry2019-11-04T09:22:28ZKlaus ZimmermannAdd option to deactivate sentryDuring development and debugging it can be useful to avoid sending sentry notes.
For this purpose it would be nice to have an option to deactivate sentry.During development and debugging it can be useful to avoid sending sentry notes.
For this purpose it would be nice to have an option to deactivate sentry.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/120Improve code style2019-11-04T09:28:58ZKlaus ZimmermannImprove code styleUp to now we have not used a code style checker consistently. This has led to a small amount of code smells.
It would be good to introduce an automatic code style checker and to improve the code accordingly.Up to now we have not used a code style checker consistently. This has led to a small amount of code smells.
It would be good to introduce an automatic code style checker and to improve the code accordingly.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/124Fix dask setup to *not* prescribe local directory2019-11-04T15:31:53ZKlaus ZimmermannFix dask setup to *not* prescribe local directoryThe dask setup for local cluster at the moment prescribes a fixed location on a ram disk.
While the use of a ram disk might be beneficial and certainly not all (HPC) file systems are equally suited for the task, the prescription of one p...The dask setup for local cluster at the moment prescribes a fixed location on a ram disk.
While the use of a ram disk might be beneficial and certainly not all (HPC) file systems are equally suited for the task, the prescription of one particular directory is to restrictive and creates problems on shared environments.
For now, let's revert to dask choosing the directory, possibly giving a configuration option later on.0.9.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/125Add single threaded dask setup2019-11-08T13:28:25ZKlaus ZimmermannAdd single threaded dask setupFor debugging purposes, it is useful to have a single-threaded way of setting up dask.For debugging purposes, it is useful to have a single-threaded way of setting up dask.0.10.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/126Improve metadata handling2019-11-11T09:30:22ZKlaus ZimmermannImprove metadata handlingCleanup dataclasses based metadata handling.Cleanup dataclasses based metadata handling.https://git.smhi.se/climix/climix/-/issues/127Deal with missing long_name in parameters2019-11-11T09:35:05ZKlaus ZimmermannDeal with missing long_name in parametersQuantity parameters are not required to have a long_name. However, at the moment a missing long_name in the index definitions raises an exception. Climix should be able to handle this without a problem.Quantity parameters are not required to have a long_name. However, at the moment a missing long_name in the index definitions raises an exception. Climix should be able to handle this without a problem.0.10.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/128Improve reporting for broken indices2019-11-11T09:38:45ZKlaus ZimmermannImprove reporting for broken indicesWith support for multiple metadata files and the increasing number of indices, it would be nice to have better reporting about problems with broken indices.With support for multiple metadata files and the increasing number of indices, it would be nice to have better reporting about problems with broken indices.0.10.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/129Better implementation of spell_length (possibly other spell based indices)2020-03-04T12:52:27ZKlaus ZimmermannBetter implementation of spell_length (possibly other spell based indices)The current implementation of spell_length relies on `da.apply_along_axis`. This is far from optimal and outright problematic when the chunking in data files is badly aligned for this use case.The current implementation of spell_length relies on `da.apply_along_axis`. This is far from optimal and outright problematic when the chunking in data files is badly aligned for this use case.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/130Error reporting in the editor component2019-11-11T14:44:09ZKlaus ZimmermannError reporting in the editor componentGenerally, errors in the table are silently ignored by `climix-editor`. This can lead to, eg, missing parameters.
It would be good to add error detection and reporting to the editor to make it a bit more robust.Generally, errors in the table are silently ignored by `climix-editor`. This can lead to, eg, missing parameters.
It would be good to add error detection and reporting to the editor to make it a bit more robust.0.10.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/131Add a changelog2019-11-13T16:38:00ZKlaus ZimmermannAdd a changelogWe should keep a changelog to keep on top of the developments and to use in the release process.We should keep a changelog to keep on top of the developments and to use in the release process.0.10.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/133remove etc/metadata.yml ix.defs that are in the autogenerated metadata2019-11-20T10:09:50ZLars Bärringremove etc/metadata.yml ix.defs that are in the autogenerated metadata0.10.0Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/134Storing of parameters is broken2019-11-20T15:08:17ZKlaus ZimmermannStoring of parameters is brokenThe storing of parameters works by converting the parameter to the output units, but
this only is appropriate if by chance the units are compatible. In other cases it crashes
climix.The storing of parameters works by converting the parameter to the output units, but
this only is appropriate if by chance the units are compatible. In other cases it crashes
climix.0.10.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/136Bug in threshold data type conversion2019-11-21T13:02:45ZLars BärringBug in threshold data type conversionFor all tested precip indices climix crashes, see below. Probably the threshold should be casted as `np.float32` already when read into climix?
```
climix -s -e -x cdd /home/sm_lbarr/PROJ/test_climix/indata/CLIMPACT2_TEST/precip_AUS_...For all tested precip indices climix crashes, see below. Probably the threshold should be casted as `np.float32` already when read into climix?
```
climix -s -e -x cdd /home/sm_lbarr/PROJ/test_climix/indata/CLIMPACT2_TEST/precip_AUS_climpact2-sampledata_gridded-obs_v1_ET-SCI_v1_1991-2010.nc
INFO:root:Loading metadata
INFO:root:Scheduler ready; starting main program.
INFO:root:Preparing indices
INFO:root:Found implementation for index_function <spell_length> from distribution <climix 0.9.0+14.g29e95e2>
INFO:root:Starting calculations for index <climix.index.Index object at 0x2ab6840234a8>
INFO:root:Building output filename
INFO:root:Preparing input data
INFO:root:Calculating index
INFO:root:Adding coord categorisation.
INFO:root:Extracting period cube
INFO:root:Preparing cube
INFO:root:Calculation took 3.6670 seconds.
Traceback (most recent call last):
File "/home/sm_lbarr/.conda/envs/climix-devel-3/bin/climix", line 11, in <module>
load_entry_point('climix', 'console_scripts', 'climix')()
File "/home/sm_lbarr/CODE/climix/climix/main.py", line 126, in main
args.output_template, args.sliced_mode)
File "/home/sm_lbarr/CODE/climix/climix/main.py", line 105, in do_main
result = index(input_data)
File "/home/sm_lbarr/CODE/climix/climix/index.py", line 23, in __call__
self.index_function.prepare(sub_cube)
File "/home/sm_lbarr/CODE/climix/climix/index_functions.py", line 200, in prepare
input_cube.standard_name)
File "/home/sm_lbarr/CODE/climix/climix/util/__init__.py", line 9, in change_units
change_pr_units(cube_or_coord, new_units, new_standard_name)
File "/home/sm_lbarr/CODE/climix/climix/util/change_pr_units.py", line 118, in change_pr_units
cube_or_coord.points *= float(conv_factor)
numpy.core._exceptions.UFuncTypeError: Cannot cast ufunc 'multiply' output from dtype('float64') to dtype('int64') with casting rule 'same_kind'
```0.10.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/137Index function refactoring2019-11-21T13:00:47ZKlaus ZimmermannIndex function refactoringThe current implementations of the index functions contain quite a bit of code duplication, for example in the prepare function for indices with thresholds, or in the selection of numpy/dask pairs for conditions and reducers.
It would b...The current implementations of the index functions contain quite a bit of code duplication, for example in the prepare function for indices with thresholds, or in the selection of numpy/dask pairs for conditions and reducers.
It would be good to reduce this duplication, perhaps using mixin classes or other means of object orientation.0.10.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/138editor broken2019-11-21T14:21:02ZLars Bärringeditor broken!95 seems to have broken the editor component:
```
(climix-devel-3) ~/CODE/climix/climix/etc >climix-editor /home/sm_lbarr/CODE/cf-index-meta/climate_indices_CV22.fods >climate_indices_CV22__2.yml
Traceback (most recent call last):
Fi...!95 seems to have broken the editor component:
```
(climix-devel-3) ~/CODE/climix/climix/etc >climix-editor /home/sm_lbarr/CODE/cf-index-meta/climate_indices_CV22.fods >climate_indices_CV22__2.yml
Traceback (most recent call last):
File "/home/sm_lbarr/.conda/envs/climix-devel-3/bin/climix-editor", line 11, in <module>
load_entry_point('climix', 'console_scripts', 'climix-editor')()
File "/home/sm_lbarr/.conda/envs/climix-devel-3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/home/sm_lbarr/.conda/envs/climix-devel-3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
return ep.load()
File "/home/sm_lbarr/.conda/envs/climix-devel-3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2443, in load
return self.resolve()
File "/home/sm_lbarr/.conda/envs/climix-devel-3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/home/sm_lbarr/CODE/climix/climix/editor/__init__.py", line 1, in <module>
from .main import main
File "/home/sm_lbarr/CODE/climix/climix/editor/main.py", line 9, in <module>
from .alttable import build_index_definitions as build_alt
File "/home/sm_lbarr/CODE/climix/climix/editor/alttable.py", line 8, in <module>
from climix.index_functions import SUPPORTED_OPERATORS, SUPPORTED_REDUCERS
ImportError: cannot import name 'SUPPORTED_OPERATORS' from 'climix.index_functions' (/home/sm_lbarr/CODE/climix/climix/index_functions/__init__.py)
```Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/139update autogenerated matadata2019-11-21T17:20:53ZLars Bärringupdate autogenerated matadata0.10.0Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/140Update changelog for release 0.10.02019-11-21T17:36:39ZKlaus ZimmermannUpdate changelog for release 0.10.00.10.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/142Summary of Results and Issues emanating from tests with EOBS192022-10-05T14:00:55ZLars BärringSummary of Results and Issues emanating from tests with EOBS19Overall timing is **very favourable** compared to CLIMPACT2, here is a summary extracted from the slurm log:
```
climix -s -e -x cdd EOBS19/pr 9300.0490 seconds.
climix -s -e -x cwd EOBS19/pr 9207.9771 seconds...Overall timing is **very favourable** compared to CLIMPACT2, here is a summary extracted from the slurm log:
```
climix -s -e -x cdd EOBS19/pr 9300.0490 seconds.
climix -s -e -x cwd EOBS19/pr 9207.9771 seconds.
climix -s -e -x r10mm EOBS19/pr 335.3579 seconds.
climix -s -e -x r20mm EOBS19/pr 324.6040 seconds.
climix -s -e -x r30mm EOBS19/pr 366.9416 seconds.
climix -s -e -x sdii EOBS19/pr 403.3451 seconds.
climix -s -e -x prcptot EOBS19/pr 387.7817 seconds.
climix -s -e -x su EOBS19/tasmax 444.3942 seconds.
climix -s -e -x id EOBS19/tasmax 328.0380 seconds.
climix -s -e -x txge30 EOBS19/tasmax 336.1205 seconds.
climix -s -e -x txge35 EOBS19/tasmax 331.1915 seconds.
climix -s -e -x txx EOBS19/tasmax 326.3819 seconds.
climix -s -e -x txn EOBS19/tasmax 330.7312 seconds.
climix -s -e -x txm EOBS19/tasmax 330.0749 seconds.
climix -s -e -x tr EOBS19/tasmin 495.1577 seconds.
climix -s -e -x fd EOBS19/tasmin 335.2957 seconds.
climix -s -e -x tnlt2 EOBS19/tasmin 338.7568 seconds.
climix -s -e -x tnltm2 EOBS19/tasmin 329.6039 seconds.
climix -s -e -x tnltm20 EOBS19/tasmin 335.5620 seconds.
climix -s -e -x tnx EOBS19/tasmin 328.9269 seconds.
climix -s -e -x tnn EOBS19/tasmin 401.6072 seconds.
climix -s -e -x tnm EOBS19/tasmin 327.6455 seconds.
climix -s -e -x tmge5 EOBS19/tasmidpoint 441.7109 seconds.
climix -s -e -x tmlt5 EOBS19/tasmidpoint 323.7007 seconds.
climix -s -e -x tmge10 EOBS19/tasmidpoint 311.0065 seconds.
climix -s -e -x tmlt10 EOBS19/tasmidpoint 321.8500 seconds.
climix -s -e -x tmn EOBS19/tasmidpoint 303.0792 seconds.
climix -s -e -x tmx EOBS19/tasmidpoint 303.8381 seconds.
climix -s -e -x tmm EOBS19/tasmidpoint 320.8438 seconds.
climix -s -e -x hddheat18 EOBS19/tasmidpoint 428.0735 seconds.
climix -s -e -x cddcold18 EOBS19/tasmidpoint 356.9644 seconds.
climix -s -e -x gddgrow10 EOBS19/tasmidpoint 344.1421 seconds.
```
1. A identified before (#129) the spell based calculations (`cdd` and `cwd`) are comparatively much slower. By monitoring the progress, the step `INFO:root:Setting up aggregation` was identified as the time consuming part. Suggestion is to update the logging to facilitate this (#143)
2. For some indices (i.e. `cdd`, `cwd`, `txx`, `txn`, `tnx`, `tnn`, `tmn`, `tmx`) it appears that masking is not correctly implemented or the mask is not correctly implemented in the input file (#144). Cf. excerpt from the slurm log:
```
INFO:root:saving empty cube
/home/sm_lbarr/.conda/envs/climix-devel-3/lib/python3.7/site-packages/iris/fileformats/netcdf.py:2038: UserWarning: Cube 'air_temperature' contains unmasked data points equal to the fill-value, 1e+20. As saved,
these points will read back as missing data. To save these as normal values, please specify a 'fill_value' keyword not equal to any valid data points.
warnings.warn(msg.format(cube.name(), fill_value))
INFO:root:opening
INFO:root:Starting with DimCoord([1950-07-02 12:00:00], bounds=[[1950-01-01 00:00:00, 1951-01-01 00:00:00]], standard_name='time', calendar='gregorian', long_name='time', var_name='time')
```
3. *Preliminary* comparisons with the reference data (produced by `CLIMPACT2`) indicate that there are some differences, but this is complicated by the masked data are included.https://git.smhi.se/climix/climix/-/issues/143Update logging messages2020-02-05T11:56:08ZLars BärringUpdate logging messagesSuggestion is to update the logging message to include the following:
* Timing info (either wall clock or run time) --- to help identify which parts are slow.
* Source file and function/module --- would help me to better understand the...Suggestion is to update the logging message to include the following:
* Timing info (either wall clock or run time) --- to help identify which parts are slow.
* Source file and function/module --- would help me to better understand the code structure.0.11.0https://git.smhi.se/climix/climix/-/issues/144Improve handling of masked data in index functions2022-12-19T12:56:26ZLars BärringImprove handling of masked data in index functionsBetter handling of masked data and/or better handling of poorly formatted input files having (supposedly) masked data (cf. #142)Better handling of masked data and/or better handling of poorly formatted input files having (supposedly) masked data (cf. #142)https://git.smhi.se/climix/climix/-/issues/145Climix currently cannot handle NGCD2020-01-24T15:59:34ZLars BärringClimix currently cannot handle NGCDclimix totally cannot handle NGCD.
* start file `/home/sm_lbarr/PROJ/test_climix/climix_run_fn_ngcd.sh`
* slurm log ` /home/sm_lbarr/PROJ/test_climix/slurm-10723857.out`
* output directory (now deleted) `/home/sm_lbarr//PROJ/test_clim...climix totally cannot handle NGCD.
* start file `/home/sm_lbarr/PROJ/test_climix/climix_run_fn_ngcd.sh`
* slurm log ` /home/sm_lbarr/PROJ/test_climix/slurm-10723857.out`
* output directory (now deleted) `/home/sm_lbarr//PROJ/test_climix/outdata/NGCD/slurm-10723857/` only contained the `dask-worker-space` subdirectory, which in turn contained several subdirectories each one holding **~150 000 (!!) files**Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/146nzero index2020-02-04T15:50:20ZKlaus Zimmermannnzero indexLars to specify yaml for the nzero index.Lars to specify yaml for the nzero index.0.12.0Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/149Clean up the master table (Libreoffice)2020-02-04T16:09:41ZLars BärringClean up the master table (Libreoffice)0.11.0Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/151Index function for nzero index2020-02-04T15:46:33ZLars BärringIndex function for nzero index0.12.0Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/152Improve input variable handling2020-02-04T14:16:46ZKlaus ZimmermannImprove input variable handlingAt the moment, all metadata is for input variables is replicated every time an index uses this input variable. Given that we only use a couple of different input variables, such as `tas`, `pr`, `tasmin`, and `tasmax`, this duplication of...At the moment, all metadata is for input variables is replicated every time an index uses this input variable. Given that we only use a couple of different input variables, such as `tas`, `pr`, `tasmin`, and `tasmax`, this duplication of metadata like cell methods and standard names is unnecessary.
To better deal with input variables and prepare for future extensions, we should add a new, separate registry of input variables and only refer to that where we need the input variables in the indices.0.11.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/155Input variables with alias names2020-02-07T13:18:30ZKlaus ZimmermannInput variables with alias namesIt is common to find input data that is perfectly acceptable but uses a different variable name.
It would be nice to be able to use this data unchanged.It is common to find input data that is perfectly acceptable but uses a different variable name.
It would be nice to be able to use this data unchanged.0.11.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/158Question: parameter long name2020-02-07T11:41:21ZLars BärringQuestion: parameter long name@a002160 I saw that there is a reasonable long name for the threshold parameter of `nzero`. Currently this have to be added manually in the yaml file. Could we devise a ways to have it already in the master table? This issue resurfaced n...@a002160 I saw that there is a reasonable long name for the threshold parameter of `nzero`. Currently this have to be added manually in the yaml file. Could we devise a ways to have it already in the master table? This issue resurfaced now when I am working on the running_statistics [and thresholded_running_statistics] index function. Here we have a parameter `window_size` of `kind: quantity` and `units: days`. And I cannot find a relevant standard name (other than `time`) that has the right canonical units. So, either we ...
* ... are stuck with adding this manually to the yaml file
* ... accept that the auxiliary coordinate will have neither standard_name nor long_name, only var_name "window_size"
* ... find a way to write this into in the master table
Any thoughts? For now, I would be reluctant to open a CF ticket to add a standard name involving anything resembling time/duration or similar...https://git.smhi.se/climix/climix/-/issues/159Discuss: Configuration (take 2)2023-02-01T14:00:13ZLars BärringDiscuss: Configuration (take 2)We have gradually organised and improved the handling of configuration and metadata into yaml files. Most recently metadata was divided into index metadata .yaml files, a separate one for index_function, and another for input variable sp...We have gradually organised and improved the handling of configuration and metadata into yaml files. Most recently metadata was divided into index metadata .yaml files, a separate one for index_function, and another for input variable specification.
We should consider to have one file collecting various overall configurations that now reside at various places in the code. Candidates might be:
* MISSVAL (now a global?)
* minimum allowed number, or proportion, of data points to calculate a percentile, which is relevant at least for index functions `percentile` and `thresholded_percentile`
* the `np.inf` and `-np.inf` used to indicate "condition not met" in index functions first/last_occurrence et al.
* "Instructions" for processing of global metadata (#36)? There are two main aspects to consider:
* basic functionality should be working before climix v.1.0 is released (cf. attempt in !72)
* progress on [iris #3325](https://github.com/SciTools/iris/issues/3325) that now has been pushed to iris v.3.1
* what schema is used for output tracking_id (cf. #16)
* ...https://git.smhi.se/climix/climix/-/issues/160remove six as we require 3.8 (;=)2020-02-21T12:04:27ZLars Bärringremove six as we require 3.8 (;=)0.13https://git.smhi.se/climix/climix/-/issues/161Update editor to deal with new master table2020-02-12T13:13:23ZKlaus ZimmermannUpdate editor to deal with new master tableIn part stemming from the discussion in #135, the master table has undergone significant changes.
The editor component of climix needs to be adapted to this.In part stemming from the discussion in #135, the master table has undergone significant changes.
The editor component of climix needs to be adapted to this.0.11.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/162Improve metadata error reporting2020-02-12T12:01:01ZKlaus ZimmermannImprove metadata error reportingThe metadata error reporting could be improved. For example, if data is missing in the provided input data, we get to know which argument is missing, but not which variables are needed.The metadata error reporting could be improved. For example, if data is missing in the provided input data, we get to know which argument is missing, but not which variables are needed.0.11.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/163Update climix to deal with new metadata characteristics2020-02-12T13:13:23ZKlaus ZimmermannUpdate climix to deal with new metadata characteristicsPartly due to #135 indices coming from the master table can now have empty `cell_methods` and parameters with `proposed_standard_names`. This has been integrated into the editor in #161, but needs to be reflected in climix itself before ...Partly due to #135 indices coming from the master table can now have empty `cell_methods` and parameters with `proposed_standard_names`. This has been integrated into the editor in #161, but needs to be reflected in climix itself before the changed definitions can be integrated.0.11.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/164Update metadata with new master table and editor2020-02-12T13:13:23ZKlaus ZimmermannUpdate metadata with new master table and editorPartly due to #135, the master table has changed significantly.
This is reflected in changes to the editor in #161 and to climix in #163.
Finally, this needs to be put into the actual metadata, ie index and variable definitions.Partly due to #135, the master table has changed significantly.
This is reflected in changes to the editor in #161 and to climix in #163.
Finally, this needs to be put into the actual metadata, ie index and variable definitions.0.11.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/165Update changelog for release 0.11.02020-02-12T13:43:57ZKlaus ZimmermannUpdate changelog for release 0.11.00.11.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/167Do not "format" standard names2020-02-19T15:46:00ZLars BärringDo not "format" standard namesSome metadata elements can be "parameterised" to include the [numeric] value instead of a placeholder within braces (e.g. `{PRC}`). However this should not be done for either `standard_name` or `proposed_standard_name`, irrespective of w...Some metadata elements can be "parameterised" to include the [numeric] value instead of a placeholder within braces (e.g. `{PRC}`). However this should not be done for either `standard_name` or `proposed_standard_name`, irrespective of whether it is the index variable or and auxiliary coordinate.
Cf. disussion in !115 for background.0.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/168Index vdtr (day-to-day variation in dtr)2020-02-20T13:23:03ZKlaus ZimmermannIndex vdtr (day-to-day variation in dtr)0.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/169Link the master table better2020-02-20T12:24:12ZKlaus ZimmermannLink the master table betterNow that we rely on the master table to create our metadata, we should make it easier to keep the two in sync.
One way would be to add this as a submodule in git and put the hash of the used master table revision into the metadata files.Now that we rely on the master table to create our metadata, we should make it easier to keep the two in sync.
One way would be to add this as a submodule in git and put the hash of the used master table revision into the metadata files.0.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/170Mechanism to specify user defined seasons in the master table2022-11-21T11:35:10ZLars BärringMechanism to specify user defined seasons in the master tableCurrently it is not possible to specify a user defined season master table. Existing use cases are the faf that is based on JASOND, and lsf that is based on JFMAMJ. Both these are "annual" indices as per discussion/conclusion/decision in...Currently it is not possible to specify a user defined season master table. Existing use cases are the faf that is based on JASOND, and lsf that is based on JFMAMJ. Both these are "annual" indices as per discussion/conclusion/decision in #95https://git.smhi.se/climix/climix/-/issues/171Update metadata to master table version 0.1.02020-02-20T13:23:03ZKlaus ZimmermannUpdate metadata to master table version 0.1.00.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/172Add changelog utility scripts2020-02-20T14:52:47ZKlaus ZimmermannAdd changelog utility scripts0.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/173Update changelog for release 0.12.02020-02-20T15:26:52ZKlaus ZimmermannUpdate changelog for release 0.12.00.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/174Housekeeping2020-02-21T10:34:27ZKlaus ZimmermannHousekeepingA number of housekeeping tasks are open:
- [x] get rid of legacy folder
- [x] get rid of legacy branch
- [x] ensure order of index functionsA number of housekeeping tasks are open:
- [x] get rid of legacy folder
- [x] get rid of legacy branch
- [x] ensure order of index functions0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/175indices based on annual climatology of percentiles2023-01-31T16:47:06ZLars Bärringindices based on annual climatology of percentilesThis is an overall issue aiming at splitting the "issue of percentile based indices" into more tangible and concrete sub-issues.
1. [x] Indices counting the number of occurrences (days) above/below a temporally and spatially variable t...This is an overall issue aiming at splitting the "issue of percentile based indices" into more tangible and concrete sub-issues.
1. [x] Indices counting the number of occurrences (days) above/below a temporally and spatially variable threshold. I.e. for each point (gridcell) there is a threshold value for each day of the year (complication: leap days!). Example indices are `tn10p, tx10p, tg10p, tx90p, tg90p, txgt50p, txgt{PRC}p, tngt{PRC}p, tmgt{PRC}p, txlt{PRC}p, tnlt{PRC}p, tmlt{PRC}p. Cf. #196 (M 0.13)
2. [x] Same as for point 1, but the climatology of threshold values is based on days meeting a threshold condition. E.g. for `r75p, r95p, r99p, r{PRC}p` only "rainy days" (e.g. days with precipitation of at least X mm day-1) are included in the percentile calculation. Cf. #197.
3. [x] Indices giving the total precipitation, in mm day-1, above a percentile threshold as a percentage of total wet-day precipitation, e.g. `r95ptot, r99ptot, r{PRC}ptot`.
4. [ ] Indices counting the total number of days belonging to temperature spells of length at least ND above/below a percentile-based threshold (as per details under 1). Example indices include `wsdi, wsdi{ND}, csdi, csdi{ND}`. In these indices the percentile threshold is fixed at 90 for wsdi and 10 for csdi, but this should be possible to specify as a parameter.
5. [ ] A few other indices involving two input variables, each one with its own threshold array. Examples are `txDtnD, ...`, but let's leave them for now as they need further specification (as to whether it is all days or only the longest spell).
-------------------
The threshold array is calculated as a daily climatology over a reference period, typically 30 years. For each day the percentile is calculated based all days in all years of the reference period and falling under a centred 5-day sliding window. This a bit involved as
* the window should extend into the previous/next year (including extending outside the reference period)
* handling of leap years
* use the right percentile method, see [wiki page](https://git.smhi.se/climix/climix/-/wikis/Memo-on-percentile-methods)
* use bootstrapping as per [Zhang et al, 2005](https://doi.org/10.1175/JCLI3366.1)
For thresholded data, e.g. indices based on "rainy days" include only RR >= 1 mm day-1 in percentile calculation.https://git.smhi.se/climix/climix/-/issues/178percentile method2020-03-13T14:53:16ZLars Bärringpercentile methodETCCDI is rather specific when it comes to percentiles, cf. [1]. Mention #75, #148, #175
The specified method, Hyndman & Fan no. 8, is available in scipy.stats.mstats.mquantiles. It would be useful to make use of it if not too detrimen...ETCCDI is rather specific when it comes to percentiles, cf. [1]. Mention #75, #148, #175
The specified method, Hyndman & Fan no. 8, is available in scipy.stats.mstats.mquantiles. It would be useful to make use of it if not too detrimental to timing. So I did a very quick test (patterned after some Stack Exchange answer):
```python
setup = '''
import random
import numpy as np
import scipy.stats.mstats as mstats
random.seed('slartibartfast')
s = [random.random() for i in range(1000)]
p = 35
third = 1./3. # this particular value gives H&F#8
numper = lambda x: np.percentile(x, p)
statper = lambda x: mstats.mquantiles(x, [p/100.], alphap=third, betap=third)
'''
print(min(timeit.Timer('x=s[:]; statper(x)', setup=setup).repeat(7, 1000)))
# 2.9134851020062342
print(min(timeit.Timer('x=s[:]; numper(x)', setup=setup).repeat(7, 1000)))
# 0.1574339639628306
```
~~This does not look overly promising, so I suggest that we leave this for now. Let's see what they might come up with over at numpy, cf. [1]~~
[1] [Wiki page](https://git.smhi.se/climix/climix/-/wikis/Memo-on-percentile-methods)https://git.smhi.se/climix/climix/-/issues/180Improve post processing infrastructure2020-04-20T13:19:51ZKlaus ZimmermannImprove post processing infrastructurePost-processing in index functions is currently limited by the requirement that the result of the calculation is stored in the final result cube already before post-processing. This has two unfortunate consequences. First, the shape of t...Post-processing in index functions is currently limited by the requirement that the result of the calculation is stored in the final result cube already before post-processing. This has two unfortunate consequences. First, the shape of the pre-result must already conform to the shape of the final result, preventing the passing of any extra information that is only used in the post-processing. Second, the pre-result must already be in the form of a single array, which can be too limiting under certain circumstances.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/181Pass dask.distributed.Client to index functions2020-04-20T13:24:05ZKlaus ZimmermannPass dask.distributed.Client to index functionsMaking the distributed client available to the index functions allows for better exploitation of dask features.Making the distributed client available to the index functions allows for better exploitation of dask features.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/182Move sliced mode to pre-result computation2020-04-20T13:35:41ZKlaus ZimmermannMove sliced mode to pre-result computationGreater flexibility in the post-processing makes it necessary to move the sliced mode to the pre-result stage.
At the same time this provides an opportunity to improve the output layer.Greater flexibility in the post-processing makes it necessary to move the sliced mode to the pre-result stage.
At the same time this provides an opportunity to improve the output layer.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/183Improve spell length calculation to allow for spells crossing period boundaries2020-04-20T13:43:39ZKlaus ZimmermannImprove spell length calculation to allow for spells crossing period boundariesUsing the new post-processing infrastructure, we should be able to implement spell length calculations that treat spells crossing period boundaries the same way as the reference implementation does.Using the new post-processing infrastructure, we should be able to implement spell length calculations that treat spells crossing period boundaries the same way as the reference implementation does.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/184SCALAR_REDUCERS is missing mean2020-04-22T09:28:25ZKlaus ZimmermannSCALAR_REDUCERS is missing meanThe `SCALAR_REDUCERS` in `climix.index_functions.support` are missing an implementation for `mean`.
This triggers an exception in the ReducerMixin and should be added.The `SCALAR_REDUCERS` in `climix.index_functions.support` are missing an implementation for `mean`.
This triggers an exception in the ReducerMixin and should be added.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/185Detect tty and adapt output2020-10-19T15:27:50ZKlaus ZimmermannDetect tty and adapt outputRecently, we added progress bars for certain kinds of calculations. This works well for ttys, ie interactive terminals, but is rather distracting in log files created by batch runs or shell redirection. Hence, it would be nice to use `sy...Recently, we added progress bars for certain kinds of calculations. This works well for ttys, ie interactive terminals, but is rather distracting in log files created by batch runs or shell redirection. Hence, it would be nice to use `sys.stdout.isatty()` to detect the way climix is being run and to produce appropriate output for the circumstances.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/186Log information on which index definitions yaml files are read2022-11-16T13:00:53ZLars BärringLog information on which index definitions yaml files are readThis seems like generally useful information, and in particular it helps if there is an error in one of the files.This seems like generally useful information, and in particular it helps if there is an error in one of the files.Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/187Change diurnal_temperature_range index function to take reducer as argument2022-11-15T15:10:53ZLars BärringChange diurnal_temperature_range index function to take reducer as argument0.15Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/188Add flag parameter kind2020-09-15T14:58:49ZKlaus ZimmermannAdd flag parameter kindSometimes it's useful to be able to switch certain behavior on or off in index calculations.
An example is the fusing of spells across period boundaries that is sometimes desired, sometimes not.
To control this from the yaml files, we ne...Sometimes it's useful to be able to switch certain behavior on or off in index calculations.
An example is the fusing of spells across period boundaries that is sometimes desired, sometimes not.
To control this from the yaml files, we need a new parameter kind.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/189Consider spells for all affected periods2020-09-15T15:11:30ZKlaus ZimmermannConsider spells for all affected periodsUp to this point, if a spell affects several periods, it is only considered for the last period.
It would be better to consider the (partial) spell for every period that it touches.Up to this point, if a spell affects several periods, it is only considered for the last period.
It would be better to consider the (partial) spell for every period that it touches.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/190[Re-]consider what information should be logged in debug mode and in info mode2022-11-16T14:53:59ZLars Bärring[Re-]consider what information should be logged in debug mode and in info modeSome of the information logged in info mode do not make much sense outside the development context, while other more user-oriented information may be missing.
Here are some examples *[with thoughts on what should be changed]*:
* INFO:...Some of the information logged in info mode do not make much sense outside the development context, while other more user-oriented information may be missing.
Here are some examples *[with thoughts on what should be changed]*:
* INFO:root:Loading metadata *[debug, cf. #186 which could be info]*
* INFO:root:Scheduler ready; starting main program. *[debug]*
* INFO:root:Preparing indices *[debug]*
* INFO:root:Found implementation for index_function <spell_length> from distribution <climix 0.12.0+3.g0ca00ff> *[debug]*
* INFO:root:Starting calculations for index <climix.index.Index object at 0x7f244480b730> *[debug]*
* INFO:root:Building output filename *[info, but give the resulting output filename pattern]*
* INFO:root:Preparing input data *[debug]*
* INFO:root:Calculation took 3.4490 seconds. *[info (as is)]*0.16https://git.smhi.se/climix/climix/-/issues/273Reference period definition in index_definitions.yml2023-04-19T15:05:32ZCarolina NilssonReference period definition in index_definitions.ymlTo conclude after offline conversation. Some index definitions utilises a reference period in the definition for the calculation and there is no obvious way in how this should be handled. A suggestion would be to included the reference p...To conclude after offline conversation. Some index definitions utilises a reference period in the definition for the calculation and there is no obvious way in how this should be handled. A suggestion would be to included the reference period in the clix-meta index definition, given the following example:
```
index_function:
name: count_thresholded_percentile_occurrences
parameters:
data_threshold:
kind: quantity
standard_name: lwe_precipitation_rate
long_name: "Wet day threshold"
data: 1
units: "mm day-1"
data_condition:
kind: operator
operator: ">="
percentile:
kind: quantity
standard_name:
proposed_standard_name: quantile
long_name: "Percentile value"
data: "{PRC}"
units: "%"
percentile_condition:
kind: operator
operator: ">"
reference_period:
kind: time_period
data: ['1961-01-01 00:00:00', '1991-01-01 00:00:00']
```
the reference period could then be used in the index calculation and the output should contain a global attribute giving information about which reference period that was used.
```
global attributes:
:reference_period = "1961-01-01 00:00:00 to 1991-01-01 00:00:00"
```
or something similar.
Notes there were also a discussion about having an additional Auxiliary coordinate variable with the following information:
```
reference_period:(var_name: "reference_period", standard_name: "reference_epoch", value: mean(n1,n2), bounds: [n1, n2], unit: "days since 1961-01-01 00:00:00")
```
where n1 and n2 are the point values for the start and end of the time period defined by the unit.0.18 (Gull Olle)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/36Transfer/process input 'global' metadata to create output 'global' metadata2023-04-19T15:05:32ZLars BärringTransfer/process input 'global' metadata to create output 'global' metadata0.18 (Gull Olle)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/275Add integration tests using NGCD dataset2023-04-21T15:56:13ZJoakim LöwAdd integration tests using NGCD datasetAdd integration tests running climix on the NGCD dataset, and compare results with NGCD reference indicators.Add integration tests running climix on the NGCD dataset, and compare results with NGCD reference indicators.0.18 (Gull Olle)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/50(Re-)Add License2023-04-19T15:05:32ZKlaus Zimmermann(Re-)Add LicenseWe had a discussion about the license already in #3.
This was in the context of what we now call legacy, but I suggest to simply take the same license.
@a001257 ok?We had a discussion about the license already in #3.
This was in the context of what we now call legacy, but I suggest to simply take the same license.
@a001257 ok?0.18 (Gull Olle)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/284Amend the yaml reader function to be aware of the CLIX-META version number2023-04-19T15:05:32ZLars BärringAmend the yaml reader function to be aware of the CLIX-META version number0.18 (Gull Olle)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/304Modernize build infrastructure2023-04-19T15:05:32ZKlaus ZimmermannModernize build infrastructureWe rely on traditional `setup.py`, `setup.cfg`, and various tool specific config files. We should follow modern Python packaging standards and unify things in `pyproject.toml`.We rely on traditional `setup.py`, `setup.cfg`, and various tool specific config files. We should follow modern Python packaging standards and unify things in `pyproject.toml`.0.18 (Gull Olle)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/252Allow coordinate "year" (e.g.) to exist already2023-11-28T14:09:22ZRenate WilckeAllow coordinate "year" (e.g.) to exist alreadyWhen the data used to calculate indices with has already a coordinate called 'year' (this might be true for other aux coordinates like "season"...) I get an error in climix
/climix/period.py", line 22, in add_coord_categorisation
I wond...When the data used to calculate indices with has already a coordinate called 'year' (this might be true for other aux coordinates like "season"...) I get an error in climix
/climix/period.py", line 22, in add_coord_categorisation
I wonder if we can not simply write a try/except here and pass if an error is thrown.0.20 (Urbane Goat)Renate WilckeRenate Wilckehttps://git.smhi.se/climix/climix/-/issues/337Place for climix logo files2023-11-06T10:13:01ZJoakim LöwPlace for climix logo files- Find good place/folder for storing the official climix logo files
- Also add climix logo as favicon- Find good place/folder for storing the official climix logo files
- Also add climix logo as favicon0.20 (Urbane Goat)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/325Error when integrations tests are generated2023-11-05T15:29:56ZJoakim LöwError when integrations tests are generatedIntegrations tests currently cannot be run. Pytest returns the following error when running from command line:
```
====================================================================== test session starts ==============================...Integrations tests currently cannot be run. Pytest returns the following error when running from command line:
```
====================================================================== test session starts ======================================================================
platform linux -- Python 3.10.11, pytest-7.3.2, pluggy-1.0.0
rootdir: /home/sm_joalo/dev/repos/climix
configfile: pyproject.toml
testpaths: tests
collected 213 items / 1 error
============================================================================ ERRORS =============================================================================
______________________________________________________ ERROR collecting tests/integration/test_indices.py _______________________________________________________
tests/integration/test_indices.py:64: in <module>
generate_test_index_parametrization(),
tests/integration/test_indices.py:45: in generate_test_index_parametrization
config = read_test_configuration()
tests/integration/conftest.py:14: in read_test_configuration
config_string = files("tests.integration").joinpath("configuration.yml").read_text()
../../../.conda/envs/climix/lib/python3.10/importlib/_common.py:22: in files
return from_package(get_package(package))
../../../.conda/envs/climix/lib/python3.10/importlib/_common.py:67: in get_package
if wrap_spec(resolved).submodule_search_locations is None:
../../../.conda/envs/climix/lib/python3.10/importlib/_adapters.py:16: in __getattr__
return getattr(self.spec, name)
E AttributeError: 'NoneType' object has no attribute 'submodule_search_locations'
==================================================================== short test summary info ====================================================================
ERROR tests/integration/test_indices.py - AttributeError: 'NoneType' object has no attribute 'submodule_search_locations'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================= 1 error in 24.99s =======================================================================
```0.20 (Urbane Goat)Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/328Index: Number of days(Temperature between two values (-2, 2) and precipitatio...2023-11-05T13:23:52ZJoakim LöwIndex: Number of days(Temperature between two values (-2, 2) and precipitation above 0.1 mm/d)Similar to index functions:
`CountJointOccurrencesPrecipitationTemperature`
`CountJointOccurrencesTemperature`
which inherits from
`CountJointOccurrences`
Replaces #296
Note: check if it's ok to add index definition in `SMHI_extra.yml`.Similar to index functions:
`CountJointOccurrencesPrecipitationTemperature`
`CountJointOccurrencesTemperature`
which inherits from
`CountJointOccurrences`
Replaces #296
Note: check if it's ok to add index definition in `SMHI_extra.yml`.0.20 (Urbane Goat)Renate WilckeRenate Wilcke