climix issueshttps://git.smhi.se/climix/climix/-/issues2023-04-19T15:05:32Zhttps://git.smhi.se/climix/climix/-/issues/294Function to parse date and timerange in iso 8601 format2023-04-19T15:05:32ZJoakim LöwFunction to parse date and timerange in iso 8601 formatDates and time ranges will be required to be given in iso 8601 format in index definition yaml (reference period, see #273) and for command line arguments (reference period and computational period, see #289).
The end date of a time ran...Dates and time ranges will be required to be given in iso 8601 format in index definition yaml (reference period, see #273) and for command line arguments (reference period and computational period, see #289).
The end date of a time range should be interpreted as the upper bound of the date, i.e. `1961/1990` should mean `1961-01-01T00:00:00` to `1991-01-01T00:00:00` and `1961-01/1990-01` should mean `1961-01-01T00:00:00` to `1990-02-01T00:00:00` (see https://git.smhi.se/climix/climix/-/issues/273#note_34689)
The time range parser should have support for duration as start or end, i.e. `P20Y/2100` should produce same as `2081/2100`.0.17 (Black Lion)Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/293Allow index functions to use coordinate information2023-02-13T15:18:49ZLars BärringAllow index functions to use coordinate informationSome indices need to include coordinate information in the calculation. Examples are:
* growing season length (`gsl`) and related indices that use latitude information to shift calculation period 6 month in the southern hemisphere
* suns...Some indices need to include coordinate information in the calculation. Examples are:
* growing season length (`gsl`) and related indices that use latitude information to shift calculation period 6 month in the southern hemisphere
* sunshine duration (`SSp`) that calculates the actual sunshine duration as a fraction of the day length (latitude, time)https://git.smhi.se/climix/climix/-/issues/292Add linting test stage using GitLab CI /Kubernetes2024-02-02T12:06:13ZJoakim LöwAdd linting test stage using GitLab CI /KubernetesFirst step utilizing Kubernetes for CI: add a linting test stage to run linting on a MR.First step utilizing Kubernetes for CI: add a linting test stage to run linting on a MR.Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/291Make use of guvectorize in spell kernels2023-02-24T07:32:52ZErik HolmgrenMake use of guvectorize in spell kernelsTracking the progress of the investigation, and possible implementation, of using `@guvectorize` for `chunk_column` in spell kernels.
Essentially, this means that we could remove the loop over the cube dimensions in `chunk` and let numb...Tracking the progress of the investigation, and possible implementation, of using `@guvectorize` for `chunk_column` in spell kernels.
Essentially, this means that we could remove the loop over the cube dimensions in `chunk` and let numba vectorize it.Erik HolmgrenErik Holmgrenhttps://git.smhi.se/climix/climix/-/issues/290Handeling of non-CMORized input data2023-11-13T12:07:05ZCarolina NilssonHandeling of non-CMORized input dataIn issue #15 there was a discussion on how climix would handle input data that is not CMORized, i.e., does not follows the CF-conventions and CMIP standards. This is moved to a separate issue since a decision needs to be taken about if ...In issue #15 there was a discussion on how climix would handle input data that is not CMORized, i.e., does not follows the CF-conventions and CMIP standards. This is moved to a separate issue since a decision needs to be taken about if we should rely on running some other tool first (ESMValTool) or if we should implement some solution on our own possibly using ESMValTools API or similar.https://git.smhi.se/climix/climix/-/issues/289Command line option to specify computational period2023-12-12T09:57:28ZJoakim LöwCommand line option to specify computational periodAdd a command line option to specify the period to perform index calculation on. Reasons:
1. User may only want to compute the index for a sub period of the data.
2. Beginning and end of input data may be incomplete with respect to time ...Add a command line option to specify the period to perform index calculation on. Reasons:
1. User may only want to compute the index for a sub period of the data.
2. Beginning and end of input data may be incomplete with respect to time period. E.g. if 'annual' and first and/or last year of data does not contain entries for all days of year, climix may raise error for some indices.
See also #273 and #2570.20 (Urbane Goat)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/288SMHI start of seasons2024-02-02T12:08:17ZLars BärringSMHI start of seasonsImplement indicators (indices) for SMHI definitions of [start of] seasons. Possibly this can be done using existing functionality (`gsstart`?) and only new entries in the yaml file.
First step is to find out exactly how the season trans...Implement indicators (indices) for SMHI definitions of [start of] seasons. Possibly this can be done using existing functionality (`gsstart`?) and only new entries in the yaml file.
First step is to find out exactly how the season transitions are defined.0.22Gustav StrandbergGustav Strandberghttps://git.smhi.se/climix/climix/-/issues/287Produce a nice summary of which indices are available in climix2023-04-23T13:41:47ZLars BärringProduce a nice summary of which indices are available in climixUsers have requested to have access, possibly published via SMHI internal web pages to which indices that climix can produce.
A starting point may be the output from `climix -x list`:
``` bash
~ >climix -x list
INFO:root:Activating sen...Users have requested to have access, possibly published via SMHI internal web pages to which indices that climix can produce.
A starting point may be the output from `climix -x list`:
``` bash
~ >climix -x list
INFO:root:Activating sentry (automatic error reporting)
INFO:root:Looking for metadata in directory /home/sm_lbarr/CODE/climix/climix/etc
INFO:root:Looking for metadata in directory /etc/climix
INFO:root:Looking for metadata in directory /home/sm_lbarr/.config/climix
INFO:root:Reading index definitions from file /home/sm_lbarr/CODE/climix/climix/etc/metadata.yml
INFO:root:Reading index definitions from file /home/sm_lbarr/CODE/climix/climix/etc/variables.yml
INFO:root:Reading index definitions from file /home/sm_lbarr/CODE/climix/climix/etc/index_definitions.yml
Available indices are:
['fd', 'tnlt2', 'tnltm2', 'tnltm20', 'id', 'su', 'txge30', 'txge35', 'tr', 'tmge5', 'tmlt5', 'tmge10', 'tmlt10',
'tngt{TT}', 'tnlt{TT}', 'tnge{TT}', 'tnle{TT}', 'txgt{TT}', 'txlt{TT}', 'txge{TT}', 'txle{TT}', 'tmgt{TT}',
'tmlt{TT}', 'tmge{TT}', 'tmle{TT}', 'ctngt{TT}', 'cfd', 'csu', 'ctnlt{TT}', 'ctnge{TT}', 'ctnle{TT}', 'ctxgt{TT}',
'ctxlt{TT}', 'ctxge{TT}', 'ctxle{TT}', 'ctmgt{TT}', 'ctmlt{TT}', 'ctmge{TT}', 'ctmle{TT}', 'txx', 'tnx', 'txn', '
tnn', 'txm', 'tnm', 'tmx', 'tmn', 'tmm', 'txmax', 'tnmax', 'txmin', 'tnmin', 'txmean', 'tnmean', 'tmmax', 'tmmin',
'tmmean', 'tn10p', 'tx10p', 'tn90p', 'tx90p', 'tg10p', 'tg90p', 'txgt50p', 'txgt{PRC}p', 'tngt{PRC}p', 'tmgt{PRC}p',
'txlt{PRC}p', 'tnlt{PRC}p', 'tmlt{PRC}p', 'dtr', 'vdtr', 'etr', 'tx{PRC}pctl', 'tn{PRC}pctl', 'tm{PRC}pctl',
'hd17', 'hddheat{TT}', 'ddgt{TT}', 'cddcold{TT}', 'ddlt{TT}', 'gddgrow{TT}', 'gd4', 'r10mm', 'r20mm', 'r{RT}mm',
'wetdays', 'rr1', 'cdd', 'cwd', 'prcptot', 'sdii', 'r{PRC}pctl', 'r{PRC}pDAYS', 'rx1day', 'rx5day', 'rx{ND}day',
'rh', 'rr', 'pp', 'tg', 'tn', 'tx', 'sd', 'sd1', 'sd5cm', 'sd50cm', 'sd{D}cm', 'ss', 'fxx', 'fg6bft', 'fgcalm',
'fg', 'nzero', 'maxdtr']
```
This can of course be expanded in various directions, e.g. to
* list the indices per file that is scanned,
* also give the `long_name`, `OUTPUT_unit`, input variable(s), ...
* see #57 for another and much more complex idea0.18 (Gull Olle)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/286Index function for CW, CD, WW, WD2023-02-27T20:14:19ZLars BärringIndex function for CW, CD, WW, WDECA&D defines these two-variable indices as:
| VarName | OUTPUT_long_name |
|---------|---------------------------------------...ECA&D defines these two-variable indices as:
| VarName | OUTPUT_long_name |
|---------|--------------------------------------------------------------------------------------------------------------------------------|
| CD | Days with TG < 25th percentile of daily mean temperature and RR < 25th percentile of daily precipitation sum ("cold/dry days") |
| CW | Days with TG < 25th percentile of daily mean temperature and RR > 75th percentile of daily precipitation sum ("cold/wet days") |
| WD | Days with TG > 75th percentile of daily mean temperature and RR < 25th percentile of daily precipitation sum ("warm/dry days") |
| WW | Days with TG > 75th percentile of daily mean temperature and RR > 75th percentile of daily precipitation sum ("warm/wet days") |
where TG is "air **T**emperature near the **G**round ", i.e. the usual daily mean temperature `tas`, and RR is daily total precipitation `pr`.https://git.smhi.se/climix/climix/-/issues/285Implementation of index function for WSDI, CSDI2023-10-26T13:13:52ZLars BärringImplementation of index function for WSDI, CSDIWSDI and CSDI are (briefly) defined by ETCCDI and ET-SCI as
|VarName | OUTPUT_long_name |
|----------|------------------------------...WSDI and CSDI are (briefly) defined by ETCCDI and ET-SCI as
|VarName | OUTPUT_long_name |
|----------|-----------------------------------------------------------------------------------------------------------------------|
| wsdi | Warm Spell Duration Index, count of days with at least 6 consecutive days when Tmax > 90th percentile |
| wsdi{ND} | User-defined Warm Spell Duration Index, count of days with at least {ND} consecutive days when Tmax > 90th percentile |
| csdi | Cold Spell Duration Index, count of days with at least 6 consecutive days when Tmin < 10th percentile |
| csdi{ND} | User-defined Cold Spell Duration Index, count of days with at least # consecutive days when Tmin < 10th percentile |https://git.smhi.se/climix/climix/-/issues/284Amend the yaml reader function to be aware of the CLIX-META version number2023-04-19T15:05:32ZLars BärringAmend the yaml reader function to be aware of the CLIX-META version number0.18 (Gull Olle)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/281Climix API2023-06-09T14:05:16ZCarolina NilssonClimix APISome users use the functions in Climix and their own scripts which can be problematic in some cases. We need to take a decision on how one should use Climix, i.e. if we should limit and test the functions and create some sort of Climix API.Some users use the functions in Climix and their own scripts which can be problematic in some cases. We need to take a decision on how one should use Climix, i.e. if we should limit and test the functions and create some sort of Climix API.https://git.smhi.se/climix/climix/-/issues/280new index request: number of events2023-09-12T08:41:46ZRenate Wilckenew index request: number of eventsI wish for an index that gives the "number of events", where an event would be (in my case).
Where event is defined as: (T > 95%) x number of day in a row. Where T can be tmean, tmax, or tmin.
Concrete example is (Tmax > 20C) at least 3 ...I wish for an index that gives the "number of events", where an event would be (in my case).
Where event is defined as: (T > 95%) x number of day in a row. Where T can be tmean, tmax, or tmin.
Concrete example is (Tmax > 20C) at least 3 days in a row.https://git.smhi.se/climix/climix/-/issues/275Add integration tests using NGCD dataset2023-04-21T15:56:13ZJoakim LöwAdd integration tests using NGCD datasetAdd integration tests running climix on the NGCD dataset, and compare results with NGCD reference indicators.Add integration tests running climix on the NGCD dataset, and compare results with NGCD reference indicators.0.18 (Gull Olle)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/273Reference period definition in index_definitions.yml2023-04-19T15:05:32ZCarolina NilssonReference period definition in index_definitions.ymlTo conclude after offline conversation. Some index definitions utilises a reference period in the definition for the calculation and there is no obvious way in how this should be handled. A suggestion would be to included the reference p...To conclude after offline conversation. Some index definitions utilises a reference period in the definition for the calculation and there is no obvious way in how this should be handled. A suggestion would be to included the reference period in the clix-meta index definition, given the following example:
```
index_function:
name: count_thresholded_percentile_occurrences
parameters:
data_threshold:
kind: quantity
standard_name: lwe_precipitation_rate
long_name: "Wet day threshold"
data: 1
units: "mm day-1"
data_condition:
kind: operator
operator: ">="
percentile:
kind: quantity
standard_name:
proposed_standard_name: quantile
long_name: "Percentile value"
data: "{PRC}"
units: "%"
percentile_condition:
kind: operator
operator: ">"
reference_period:
kind: time_period
data: ['1961-01-01 00:00:00', '1991-01-01 00:00:00']
```
the reference period could then be used in the index calculation and the output should contain a global attribute giving information about which reference period that was used.
```
global attributes:
:reference_period = "1961-01-01 00:00:00 to 1991-01-01 00:00:00"
```
or something similar.
Notes there were also a discussion about having an additional Auxiliary coordinate variable with the following information:
```
reference_period:(var_name: "reference_period", standard_name: "reference_epoch", value: mean(n1,n2), bounds: [n1, n2], unit: "days since 1961-01-01 00:00:00")
```
where n1 and n2 are the point values for the start and end of the time period defined by the unit.0.18 (Gull Olle)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/272tn10p, tn90p, tx10p, tx90p with NGCD data results in ZeroDivisionError2023-01-30T14:51:02ZJoakim Löwtn10p, tn90p, tx10p, tx90p with NGCD data results in ZeroDivisionErrorCalculating `tn10p`, `tn90p`, `tx10p` and `tx90p` with input from NGCD dataset results in a `ZeroDivisionError`. E.g.:
```
climix -v -e -x tn10p /home/rossby/joint_exp/climix/1/testdata/NGCD/input_data/SverigeScaled/NGCD_TN_type2_*.nc
``...Calculating `tn10p`, `tn90p`, `tx10p` and `tx90p` with input from NGCD dataset results in a `ZeroDivisionError`. E.g.:
```
climix -v -e -x tn10p /home/rossby/joint_exp/climix/1/testdata/NGCD/input_data/SverigeScaled/NGCD_TN_type2_*.nc
```
generates the following traceback:
```
Traceback (most recent call last):
File "/home/sm_joalo/.conda/envs/climix-devel/bin/climix", line 8, in <module>
sys.exit(main())
File "/home/sm_joalo/dev/repos/climix/climix/main.py", line 293, in main
do_main(
File "/home/sm_joalo/dev/repos/climix/climix/main.py", line 271, in do_main
result = index(input_data, client=scheduler.client, sliced_mode=sliced_mode)
File "/home/sm_joalo/dev/repos/climix/climix/index.py", line 24, in __call__
self.index_function.preprocess(cubes, client)
File "/home/sm_joalo/dev/repos/climix/climix/index_functions/percentile_functions.py", line 232, in preprocess
all_data = all_data.rechunk(("auto",) * (all_data.ndim - 1) + (-1,))
File "/home/sm_joalo/.conda/envs/climix-devel/lib/python3.10/site-packages/dask/array/core.py", line 2745, in rechunk
return rechunk(self, chunks, threshold, block_size_limit, balance)
File "/home/sm_joalo/.conda/envs/climix-devel/lib/python3.10/site-packages/dask/array/rechunk.py", line 297, in rechunk
chunks = normalize_chunks(
File "/home/sm_joalo/.conda/envs/climix-devel/lib/python3.10/site-packages/dask/array/core.py", line 3073, in normalize_chunks
chunks = auto_chunks(chunks, shape, limit, dtype, previous_chunks)
File "/home/sm_joalo/.conda/envs/climix-devel/lib/python3.10/site-packages/dask/array/core.py", line 3206, in auto_chunks
multiplier = _compute_multiplier(limit, dtype, largest_block, result)
File "/home/sm_joalo/.conda/envs/climix-devel/lib/python3.10/site-packages/dask/array/core.py", line 3119, in _compute_multiplier
limit
ZeroDivisionError: float division by zero
```Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/271rx5day index fails with NGCD input2023-01-30T14:51:14ZJoakim Löwrx5day index fails with NGCD inputRunning the following command line:
```
climix -v -e -x rx5day /home/rossby/joint_exp/climix/1/testdata/NGCD/input_data/SverigeScaled/NGCD_RR_type2_*.nc
```
results in the following traceback:
```
Traceback (most recent call last):
Fil...Running the following command line:
```
climix -v -e -x rx5day /home/rossby/joint_exp/climix/1/testdata/NGCD/input_data/SverigeScaled/NGCD_RR_type2_*.nc
```
results in the following traceback:
```
Traceback (most recent call last):
File "/home/sm_joalo/.conda/envs/climix-devel/bin/climix", line 8, in <module>
sys.exit(main())
File "/home/sm_joalo/dev/repos/climix/climix/main.py", line 293, in main
do_main(
File "/home/sm_joalo/dev/repos/climix/climix/main.py", line 271, in do_main
result = index(input_data, client=scheduler.client, sliced_mode=sliced_mode)
File "/home/sm_joalo/dev/repos/climix/climix/index.py", line 52, in __call__
aggregated = multicube_aggregated_by(
File "/home/sm_joalo/dev/repos/climix/climix/iris.py", line 126, in multicube_aggregated_by
result = list(map(agg, groupby_subcubes))
File "/home/sm_joalo/dev/repos/climix/climix/iris.py", line 109, in agg
result = aggregate(data, axis=dimension_to_groupby, cube=cube, **kwargs)
File "/home/sm_joalo/.conda/envs/climix-devel/lib/python3.10/site-packages/iris/analysis/__init__.py", line 549, in lazy_aggregate
return self.lazy_func(data, axis=axis, **kwargs)
File "/home/sm_joalo/dev/repos/climix/climix/index_functions/index_functions.py", line 449, in lazy_func
return super().call_func(thresholded_data, axis, **kwargs)
File "/home/sm_joalo/dev/repos/climix/climix/index_functions/index_functions.py", line 354, in call_func
rolling_view = np.lib.stride_tricks.sliding_window_view(
File "<__array_function__ internals>", line 180, in sliding_window_view
File "/home/sm_joalo/.conda/envs/climix-devel/lib/python3.10/site-packages/dask/array/core.py", line 1760, in __array_function__
return da_func(*args, **kwargs)
File "/home/sm_joalo/.conda/envs/climix-devel/lib/python3.10/site-packages/dask/array/overlap.py", line 815, in sliding_window_view
safe_chunks = tuple(
File "/home/sm_joalo/.conda/envs/climix-devel/lib/python3.10/site-packages/dask/array/overlap.py", line 816, in <genexpr>
ensure_minimum_chunksize(d + 1, c) for d, c in zip(depths, x.chunks)
File "/home/sm_joalo/.conda/envs/climix-devel/lib/python3.10/site-packages/dask/array/overlap.py", line 353, in ensure_minimum_chunksize
raise ValueError(
ValueError: The overlapping depth 5 is larger than your array 1.
```Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/270Add a warning and raise a error if datafiles from unrelated datasets are used...2023-05-29T09:09:58ZCarolina NilssonAdd a warning and raise a error if datafiles from unrelated datasets are used together, e.g. tasmin and TNWhen running Climix with two input files from different datasets that uses two different variable names, e.g., tasmin and TN. Climix creates two separate cubes (which is correct), however, Climix only runs the index computation on one of...When running Climix with two input files from different datasets that uses two different variable names, e.g., tasmin and TN. Climix creates two separate cubes (which is correct), however, Climix only runs the index computation on one of them. This can be misleading for the user and go unnoticed. To improve the handling of this case a warning followed by aborting immediately should happen IF there are more than one cube after the concatenation AND if there is only one data variable required for the index calculation.https://git.smhi.se/climix/climix/-/issues/269Update clix-meta files to clix-meta release 0.4.12022-12-09T13:30:11ZJoakim LöwUpdate clix-meta files to clix-meta release 0.4.1The YAML files from Clix-meta need to be updated to the ones in the latest Clix-meta (0.4.1), which contains a fix for a bug that caused some indices to get the wrong condition operator.The YAML files from Clix-meta need to be updated to the ones in the latest Clix-meta (0.4.1), which contains a fix for a bug that caused some indices to get the wrong condition operator.0.16https://git.smhi.se/climix/climix/-/issues/268test for vegetation indices (gsl, gsstart, gsend)2023-11-07T10:38:37ZLars Bärringtest for vegetation indices (gsl, gsstart, gsend)The index for growing season length (`gsl`) is defined, but there is no "formal index" for its start (`gsstart`) and end (`gsend`). But these three indices can be calculated with CLIMIX.
Now some question marks about *gsend* have [arisen...The index for growing season length (`gsl`) is defined, but there is no "formal index" for its start (`gsstart`) and end (`gsend`). But these three indices can be calculated with CLIMIX.
Now some question marks about *gsend* have [arisen](#249). Among other things, it seems that *gsend* could occur before *gsstart*, as well as that we get `-1` as an answer in a number of more or less pathological situations.
For these three indices we would therefore need to run a number of simple test cases based on a single synthetic time series covering one or a few years.
Both *gsstart* and *gsend* use the index_function `first_spell` [=FirstSpell], while *gsl* uses `season_length` [=SeasonLength]. In this, the start date for *gsend* (July 1) has been simplified to day number 182.
Suggestions:
* The user creates an numpy array (length=365, or small multiple thereof) with synthetic data.
* A function converts this to a climix-compatible data cube, after which the three indices are calculated and the answers are returned to the user.
yaml file with index definitions can be found here:
/home/sm_klazi/.config/climix/test_indices.yml
```
GSL, Growing season length: Annual (1st Jan to 31st Dec in Northern Hemisphere (NH),
1st July to 30th June in Southern Hemisphere (SH)) count between first span of
at least 6 days with daily mean temperature TG>5°C and first span after July 1st
(Jan 1st in SH) of 6 days with TG<5°C.
Let TG_ij be daily mean temperature on day i in year j. Count the number of days between
the first occurrence of at least 6 consecutive days with:
TG_ij > 5°C
and the first occurrence after 1st July (1st Jan. in SH) of at least 6 consecutive days with:
TG_ij < 5°C.
```0.20 (Urbane Goat)Carolina NilssonCarolina Nilsson