climix issueshttps://git.smhi.se/climix/climix/-/issues2020-04-20T13:19:51Zhttps://git.smhi.se/climix/climix/-/issues/180Improve post processing infrastructure2020-04-20T13:19:51ZKlaus ZimmermannImprove post processing infrastructurePost-processing in index functions is currently limited by the requirement that the result of the calculation is stored in the final result cube already before post-processing. This has two unfortunate consequences. First, the shape of t...Post-processing in index functions is currently limited by the requirement that the result of the calculation is stored in the final result cube already before post-processing. This has two unfortunate consequences. First, the shape of the pre-result must already conform to the shape of the final result, preventing the passing of any extra information that is only used in the post-processing. Second, the pre-result must already be in the form of a single array, which can be too limiting under certain circumstances.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/179tasmidpoint as standin for tas2023-04-04T21:15:34ZLars Bärringtasmidpoint as standin for tasThe well established indices were developed within the observational data communities. And in that context daily mean temperature, as in `tas: cell_method = "time: mean"`, brings about a lot of complications (given the limited and shifti...The well established indices were developed within the observational data communities. And in that context daily mean temperature, as in `tas: cell_method = "time: mean"`, brings about a lot of complications (given the limited and shifting observation times). Hence they settled for the draconian solution that **daily mean temperature is defined as `(TN+TX)/2`**, which in terms of CF would be `tas: cell_method = "time: midpoint` (assuming that the time bound specify daily data!), where *`tas`* should have another name, such as `tasmidpoint`.
We should have the possibility to compute `tas`-based indices on `tasmidpoint`, which then should be calculated on the fly as if input data had been specified as `low_data: tasmin, high_data: tasmax`https://git.smhi.se/climix/climix/-/issues/178percentile method2020-03-13T14:53:16ZLars Bärringpercentile methodETCCDI is rather specific when it comes to percentiles, cf. [1]. Mention #75, #148, #175
The specified method, Hyndman & Fan no. 8, is available in scipy.stats.mstats.mquantiles. It would be useful to make use of it if not too detrimen...ETCCDI is rather specific when it comes to percentiles, cf. [1]. Mention #75, #148, #175
The specified method, Hyndman & Fan no. 8, is available in scipy.stats.mstats.mquantiles. It would be useful to make use of it if not too detrimental to timing. So I did a very quick test (patterned after some Stack Exchange answer):
```python
setup = '''
import random
import numpy as np
import scipy.stats.mstats as mstats
random.seed('slartibartfast')
s = [random.random() for i in range(1000)]
p = 35
third = 1./3. # this particular value gives H&F#8
numper = lambda x: np.percentile(x, p)
statper = lambda x: mstats.mquantiles(x, [p/100.], alphap=third, betap=third)
'''
print(min(timeit.Timer('x=s[:]; statper(x)', setup=setup).repeat(7, 1000)))
# 2.9134851020062342
print(min(timeit.Timer('x=s[:]; numper(x)', setup=setup).repeat(7, 1000)))
# 0.1574339639628306
```
~~This does not look overly promising, so I suggest that we leave this for now. Let's see what they might come up with over at numpy, cf. [1]~~
[1] [Wiki page](https://git.smhi.se/climix/climix/-/wikis/Memo-on-percentile-methods)https://git.smhi.se/climix/climix/-/issues/177Counting exceedance above/below threshold across differernt datasets.2023-02-13T15:14:03ZLars BärringCounting exceedance above/below threshold across differernt datasets.We have numerous indices counting exceedance above/below a fixed threshold. There is need to be able to carry out such calculations across different datasets that may have different reference climate (i.e. there are "biases"...). In a si...We have numerous indices counting exceedance above/below a fixed threshold. There is need to be able to carry out such calculations across different datasets that may have different reference climate (i.e. there are "biases"...). In a simplistic way, and without invoking bias adjustment of the input data, this could be done by carrying out a quantile mapping of the threshold applied to the reference data. For this we new a couple of new index functions
We need a couple of new functions:
1. index function to calculate a percentile rank based on a fixed threshold, e.g. `tx{TT}rank` etc. This have to be done over some reference period, typically a 30 year period like 1971-2000.
2. index function to calculate a percentile value based on **spatially varying** percentiles (e.g. produced by 1.). This have to be done over a reference period, which typically is the same as used in 1.
3. index function to count the number of exceedances (above/below) a **spatially varying** threshold (e.g. produced by 2.)
In step 2 it should be possible to read the percentile array from the file produced by 1.
Step 2 and 3 could be combined. Irrespective of this, the result of 2 should be an auxiliary coordinate in step 3.https://git.smhi.se/climix/climix/-/issues/176advanced drought related indices2023-04-27T07:43:55ZLars Bärringadvanced drought related indicesThe indices SPI and SPEI are included in the ET-SCI list, but has so far been beyond the horizon. However I just saw a [python implementation](https://github.com/monocongo/climate_indices) that seems to be solid enough in terms of instit...The indices SPI and SPEI are included in the ET-SCI list, but has so far been beyond the horizon. However I just saw a [python implementation](https://github.com/monocongo/climate_indices) that seems to be solid enough in terms of institutional support and interesting for their approach towards large data efficiency. @a002160 have a look at some point -- something we could benefit from hooking up to? The other indices on their list are well established and widely used, but typically not considered as "indices in sense where we are coming from".Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/175indices based on annual climatology of percentiles2023-01-31T16:47:06ZLars Bärringindices based on annual climatology of percentilesThis is an overall issue aiming at splitting the "issue of percentile based indices" into more tangible and concrete sub-issues.
1. [x] Indices counting the number of occurrences (days) above/below a temporally and spatially variable t...This is an overall issue aiming at splitting the "issue of percentile based indices" into more tangible and concrete sub-issues.
1. [x] Indices counting the number of occurrences (days) above/below a temporally and spatially variable threshold. I.e. for each point (gridcell) there is a threshold value for each day of the year (complication: leap days!). Example indices are `tn10p, tx10p, tg10p, tx90p, tg90p, txgt50p, txgt{PRC}p, tngt{PRC}p, tmgt{PRC}p, txlt{PRC}p, tnlt{PRC}p, tmlt{PRC}p. Cf. #196 (M 0.13)
2. [x] Same as for point 1, but the climatology of threshold values is based on days meeting a threshold condition. E.g. for `r75p, r95p, r99p, r{PRC}p` only "rainy days" (e.g. days with precipitation of at least X mm day-1) are included in the percentile calculation. Cf. #197.
3. [x] Indices giving the total precipitation, in mm day-1, above a percentile threshold as a percentage of total wet-day precipitation, e.g. `r95ptot, r99ptot, r{PRC}ptot`.
4. [ ] Indices counting the total number of days belonging to temperature spells of length at least ND above/below a percentile-based threshold (as per details under 1). Example indices include `wsdi, wsdi{ND}, csdi, csdi{ND}`. In these indices the percentile threshold is fixed at 90 for wsdi and 10 for csdi, but this should be possible to specify as a parameter.
5. [ ] A few other indices involving two input variables, each one with its own threshold array. Examples are `txDtnD, ...`, but let's leave them for now as they need further specification (as to whether it is all days or only the longest spell).
-------------------
The threshold array is calculated as a daily climatology over a reference period, typically 30 years. For each day the percentile is calculated based all days in all years of the reference period and falling under a centred 5-day sliding window. This a bit involved as
* the window should extend into the previous/next year (including extending outside the reference period)
* handling of leap years
* use the right percentile method, see [wiki page](https://git.smhi.se/climix/climix/-/wikis/Memo-on-percentile-methods)
* use bootstrapping as per [Zhang et al, 2005](https://doi.org/10.1175/JCLI3366.1)
For thresholded data, e.g. indices based on "rainy days" include only RR >= 1 mm day-1 in percentile calculation.https://git.smhi.se/climix/climix/-/issues/174Housekeeping2020-02-21T10:34:27ZKlaus ZimmermannHousekeepingA number of housekeeping tasks are open:
- [x] get rid of legacy folder
- [x] get rid of legacy branch
- [x] ensure order of index functionsA number of housekeeping tasks are open:
- [x] get rid of legacy folder
- [x] get rid of legacy branch
- [x] ensure order of index functions0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/173Update changelog for release 0.12.02020-02-20T15:26:52ZKlaus ZimmermannUpdate changelog for release 0.12.00.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/172Add changelog utility scripts2020-02-20T14:52:47ZKlaus ZimmermannAdd changelog utility scripts0.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/171Update metadata to master table version 0.1.02020-02-20T13:23:03ZKlaus ZimmermannUpdate metadata to master table version 0.1.00.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/170Mechanism to specify user defined seasons in the master table2022-11-21T11:35:10ZLars BärringMechanism to specify user defined seasons in the master tableCurrently it is not possible to specify a user defined season master table. Existing use cases are the faf that is based on JASOND, and lsf that is based on JFMAMJ. Both these are "annual" indices as per discussion/conclusion/decision in...Currently it is not possible to specify a user defined season master table. Existing use cases are the faf that is based on JASOND, and lsf that is based on JFMAMJ. Both these are "annual" indices as per discussion/conclusion/decision in #95https://git.smhi.se/climix/climix/-/issues/169Link the master table better2020-02-20T12:24:12ZKlaus ZimmermannLink the master table betterNow that we rely on the master table to create our metadata, we should make it easier to keep the two in sync.
One way would be to add this as a submodule in git and put the hash of the used master table revision into the metadata files.Now that we rely on the master table to create our metadata, we should make it easier to keep the two in sync.
One way would be to add this as a submodule in git and put the hash of the used master table revision into the metadata files.0.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/168Index vdtr (day-to-day variation in dtr)2020-02-20T13:23:03ZKlaus ZimmermannIndex vdtr (day-to-day variation in dtr)0.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/167Do not "format" standard names2020-02-19T15:46:00ZLars BärringDo not "format" standard namesSome metadata elements can be "parameterised" to include the [numeric] value instead of a placeholder within braces (e.g. `{PRC}`). However this should not be done for either `standard_name` or `proposed_standard_name`, irrespective of w...Some metadata elements can be "parameterised" to include the [numeric] value instead of a placeholder within braces (e.g. `{PRC}`). However this should not be done for either `standard_name` or `proposed_standard_name`, irrespective of whether it is the index variable or and auxiliary coordinate.
Cf. disussion in !115 for background.0.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/166index function extreme_temperature_range (index etr)2020-02-20T13:21:25ZLars Bärringindex function extreme_temperature_range (index etr)0.12.0Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/165Update changelog for release 0.11.02020-02-12T13:43:57ZKlaus ZimmermannUpdate changelog for release 0.11.00.11.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/164Update metadata with new master table and editor2020-02-12T13:13:23ZKlaus ZimmermannUpdate metadata with new master table and editorPartly due to #135, the master table has changed significantly.
This is reflected in changes to the editor in #161 and to climix in #163.
Finally, this needs to be put into the actual metadata, ie index and variable definitions.Partly due to #135, the master table has changed significantly.
This is reflected in changes to the editor in #161 and to climix in #163.
Finally, this needs to be put into the actual metadata, ie index and variable definitions.0.11.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/163Update climix to deal with new metadata characteristics2020-02-12T13:13:23ZKlaus ZimmermannUpdate climix to deal with new metadata characteristicsPartly due to #135 indices coming from the master table can now have empty `cell_methods` and parameters with `proposed_standard_names`. This has been integrated into the editor in #161, but needs to be reflected in climix itself before ...Partly due to #135 indices coming from the master table can now have empty `cell_methods` and parameters with `proposed_standard_names`. This has been integrated into the editor in #161, but needs to be reflected in climix itself before the changed definitions can be integrated.0.11.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/162Improve metadata error reporting2020-02-12T12:01:01ZKlaus ZimmermannImprove metadata error reportingThe metadata error reporting could be improved. For example, if data is missing in the provided input data, we get to know which argument is missing, but not which variables are needed.The metadata error reporting could be improved. For example, if data is missing in the provided input data, we get to know which argument is missing, but not which variables are needed.0.11.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/161Update editor to deal with new master table2020-02-12T13:13:23ZKlaus ZimmermannUpdate editor to deal with new master tableIn part stemming from the discussion in #135, the master table has undergone significant changes.
The editor component of climix needs to be adapted to this.In part stemming from the discussion in #135, the master table has undergone significant changes.
The editor component of climix needs to be adapted to this.0.11.0Klaus ZimmermannKlaus Zimmermann