climix issueshttps://git.smhi.se/climix/climix/-/issues2020-04-20T13:19:51Zhttps://git.smhi.se/climix/climix/-/issues/180Improve post processing infrastructure2020-04-20T13:19:51ZKlaus ZimmermannImprove post processing infrastructurePost-processing in index functions is currently limited by the requirement that the result of the calculation is stored in the final result cube already before post-processing. This has two unfortunate consequences. First, the shape of t...Post-processing in index functions is currently limited by the requirement that the result of the calculation is stored in the final result cube already before post-processing. This has two unfortunate consequences. First, the shape of the pre-result must already conform to the shape of the final result, preventing the passing of any extra information that is only used in the post-processing. Second, the pre-result must already be in the form of a single array, which can be too limiting under certain circumstances.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/174Housekeeping2020-02-21T10:34:27ZKlaus ZimmermannHousekeepingA number of housekeeping tasks are open:
- [x] get rid of legacy folder
- [x] get rid of legacy branch
- [x] ensure order of index functionsA number of housekeeping tasks are open:
- [x] get rid of legacy folder
- [x] get rid of legacy branch
- [x] ensure order of index functions0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/160remove six as we require 3.8 (;=)2020-02-21T12:04:27ZLars Bärringremove six as we require 3.8 (;=)0.13https://git.smhi.se/climix/climix/-/issues/129Better implementation of spell_length (possibly other spell based indices)2020-03-04T12:52:27ZKlaus ZimmermannBetter implementation of spell_length (possibly other spell based indices)The current implementation of spell_length relies on `da.apply_along_axis`. This is far from optimal and outright problematic when the chunking in data files is badly aligned for this use case.The current implementation of spell_length relies on `da.apply_along_axis`. This is far from optimal and outright problematic when the chunking in data files is badly aligned for this use case.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/101Gitlab license upgrade2020-01-21T15:38:10ZLars BärringGitlab license upgradeInitiate dialogue to support TFO/Utveckling (Cecile) to upgrade Gitlab licene to get better functionalityInitiate dialogue to support TFO/Utveckling (Cecile) to upgrade Gitlab licene to get better functionality0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/74Allow spells to extend beyond the actual calculation period (as in CLIMPACT2)2020-09-22T08:56:18ZLars BärringAllow spells to extend beyond the actual calculation period (as in CLIMPACT2)cf. #72cf. #720.13https://git.smhi.se/climix/climix/-/issues/294Function to parse date and timerange in iso 8601 format2023-04-19T15:05:32ZJoakim LöwFunction to parse date and timerange in iso 8601 formatDates and time ranges will be required to be given in iso 8601 format in index definition yaml (reference period, see #273) and for command line arguments (reference period and computational period, see #289).
The end date of a time ran...Dates and time ranges will be required to be given in iso 8601 format in index definition yaml (reference period, see #273) and for command line arguments (reference period and computational period, see #289).
The end date of a time range should be interpreted as the upper bound of the date, i.e. `1961/1990` should mean `1961-01-01T00:00:00` to `1991-01-01T00:00:00` and `1961-01/1990-01` should mean `1961-01-01T00:00:00` to `1990-02-01T00:00:00` (see https://git.smhi.se/climix/climix/-/issues/273#note_34689)
The time range parser should have support for duration as start or end, i.e. `P20Y/2100` should produce same as `2081/2100`.0.17 (Black Lion)Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/283New indicator Tmin>0 P=>12023-04-19T15:05:32ZGustav StrandbergNew indicator Tmin>0 P=>1Number of days with plus degrees and precipitation.
Since we are only working with precipitation (not just rain) it makes sense to use Tmin to reduce the number of days with snow an plus degrees as much as possible.
Question: Is it ok ...Number of days with plus degrees and precipitation.
Since we are only working with precipitation (not just rain) it makes sense to use Tmin to reduce the number of days with snow an plus degrees as much as possible.
Question: Is it ok to set the threshold of precipitation at 1 mm/day (not zero) in the same way as we do for e.g. precipitation days?0.17 (Black Lion)Carolina NilssonCarolina Nilsson2023-02-28https://git.smhi.se/climix/climix/-/issues/282New indicator 13<Tx<182023-04-19T15:05:32ZGustav StrandbergNew indicator 13<Tx<18The number of days with Tmax above 13°C and below 18°C (13 < Tx < 18). Not sure what the short name will be.The number of days with Tmax above 13°C and below 18°C (13 < Tx < 18). Not sure what the short name will be.0.17 (Black Lion)Carolina NilssonCarolina Nilsson2023-02-28https://git.smhi.se/climix/climix/-/issues/267Difference in output when storing iterative using flag -i and storing non-ite...2023-03-13T10:19:37ZCarolina NilssonDifference in output when storing iterative using flag -i and storing non-iterative on masked input dataThere is an difference in the output using iterative-storage and non-iterative storage, i.e., with or without the flag -i, on masked input data.
E.g.,
`climix -e -v -i -o /path/to/output/file.nc /home/rossby/joint_exp/climix/1/testdata...There is an difference in the output using iterative-storage and non-iterative storage, i.e., with or without the flag -i, on masked input data.
E.g.,
`climix -e -v -i -o /path/to/output/file.nc /home/rossby/joint_exp/climix/1/testdata/NGCD/annual/Aland/NGCD_RR*.nc -x r10mm`
In the output the masked area is filled with value 0. Whereas, for
`climix -e -v -o /path/to/output/file.nc /home/rossby/joint_exp/climix/1/testdata/NGCD/annual/Aland/NGCD_RR*.nc -x r10mm`
the masked area is still masked, i.e, filled with 1e+20.
Loading the output cubes and looking at the data, there seems to be a difference where one has a defined mask with shape (49, 75, 56) and the other one has the default mask=False with shape ().0.17 (Black Lion)Erik HolmgrenErik Holmgrenhttps://git.smhi.se/climix/climix/-/issues/257Add option to define a computational and reference period for indices that us...2023-03-13T10:20:34ZCarolina NilssonAdd option to define a computational and reference period for indices that uses data for reference.The computation for some indices e.g., count percentile occurrences, can be costly when the bootstrapping method is used. Right now there is no way of defining the computational period for the index such that the bootstrapping method wil...The computation for some indices e.g., count percentile occurrences, can be costly when the bootstrapping method is used. Right now there is no way of defining the computational period for the index such that the bootstrapping method will always be used. To make it possible for the user to define both a reference period and a computational period it would enable the method to run without using the bootstrapping method.
E.g., the user have a data file with time range 2001-2100 and wants to use 2001-2031 as the reference period and 2032-2100 for the index computation. The user would then define reference period to "2001-2031" and computational period to "2032-2100". The index function then needs to check if there is any overlap between these two periods because for those years the bootstrapping method would be needed. If there is no overlap the bootstrapping method is not needed. However, defining a computational period may need other changes as well such as the output template name.0.17 (Black Lion)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/256Consecutive index fails on 1 dimensional cubes2023-03-13T09:32:26ZErik HolmgrenConsecutive index fails on 1 dimensional cubesThe index `ctxge{TT}` fails on 1-dimensional cubes (i.e. cubes with only a time dimension, spatial averages etc.). Klaus did some digging and we found that the `spell_kernel` expects a 3-dimensional cube. A quick fix is to add dummy coor...The index `ctxge{TT}` fails on 1-dimensional cubes (i.e. cubes with only a time dimension, spatial averages etc.). Klaus did some digging and we found that the `spell_kernel` expects a 3-dimensional cube. A quick fix is to add dummy coordinates to such a cube before computing the index, but it would still be good if these types of indices worked on all cubes.
I haven't tried all the consecutive indices but I assume this issue is true for all of them. To get the true "exception" trying this, one has to disable numba in the `spell_kernel`.0.17 (Black Lion)Erik HolmgrenErik Holmgrenhttps://git.smhi.se/climix/climix/-/issues/193Non-CF cell methods should be empty2023-04-19T15:05:32ZLars BärringNon-CF cell methods should be emptyThis is an intermediate solution for version 1, waiting for the more thorough implementation in #52This is an intermediate solution for version 1, waiting for the more thorough implementation in #520.17 (Black Lion)Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/15Add better inconsistency reporting on input data2023-03-13T10:25:01ZKlaus ZimmermannAdd better inconsistency reporting on input dataIf the concatenation of the input data fails, it can be difficult to figure out what the problem is.
It would be nice to make it easy to identify the problems.
The aim here is to provide a catchall informative "error message" that expla...If the concatenation of the input data fails, it can be difficult to figure out what the problem is.
It would be nice to make it easy to identify the problems.
The aim here is to provide a catchall informative "error message" that explains why the data could not be concatenated. I.e. to avoid the very uninformative error message that Iris (at least in 2.2) issues.0.17 (Black Lion)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/306Update clix-meta files to v0.5.2 when ready2023-04-21T15:56:13ZLars BärringUpdate clix-meta files to v0.5.2 when readyThe new maintenance version of clix-meta will soon be ready, and it would be good to have it in Climix v.0.18.The new maintenance version of clix-meta will soon be ready, and it would be good to have it in Climix v.0.18.0.18 (Gull Olle)Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/304Modernize build infrastructure2023-04-19T15:05:32ZKlaus ZimmermannModernize build infrastructureWe rely on traditional `setup.py`, `setup.cfg`, and various tool specific config files. We should follow modern Python packaging standards and unify things in `pyproject.toml`.We rely on traditional `setup.py`, `setup.cfg`, and various tool specific config files. We should follow modern Python packaging standards and unify things in `pyproject.toml`.0.18 (Gull Olle)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/301Running climix 0.17 with new env throws hdf5 error messages2023-04-19T13:24:26ZCarolina NilssonRunning climix 0.17 with new env throws hdf5 error messagesCreating a new env using climix 0.17 environment.yml and running climix throws hdf5 error messages.
```
HDF5-DIAG: Error detected in HDF5 (1.14.0) thread 1:
#000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attrib...Creating a new env using climix 0.17 environment.yml and running climix throws hdf5 error messages.
```
HDF5-DIAG: Error detected in HDF5 (1.14.0) thread 1:
#000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute
major: Attribute
minor: Can't open object
#001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeBitRoundNumberOfSignificantBits'
major: Attribute
minor: Can't open object
#002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeBitRoundNumberOfSignificantBits'
major: Attribute
minor: Can't open object
#003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#005: H5VLnative_attr.c line 161 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#006: H5Aint.c line 658 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#007: H5Oattribute.c line 502 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'
major: Attribute
minor: Object not found
.
.
.
```
This can be solved by adding a constraint in the environment.yml `libnetcdf<4.9.1` Which makes the following changes:
```
──────────────────────────────────────────────────────────────────────────────────────
Install:
──────────────────────────────────────────────────────────────────────────────────────
+ jpeg 9e h166bdaf_2 conda-forge/linux-64 Cached
Change:
──────────────────────────────────────────────────────────────────────────────────────
- hdf4 4.2.15 h501b40f_6 conda-forge
+ hdf4 4.2.15 h9772cbc_5 conda-forge/linux-64 Cached
- lcms2 2.15 haa2dc70_1 conda-forge
+ lcms2 2.15 hfd0df8a_0 conda-forge/linux-64 Cached
- libtiff 4.5.0 ha587672_6 conda-forge
+ libtiff 4.5.0 h6adf6a1_2 conda-forge/linux-64 Cached
- pillow 9.4.0 py310h065c6d2_2 conda-forge
+ pillow 9.4.0 py310h023d228_1 conda-forge/linux-64 Cached
Downgrade:
──────────────────────────────────────────────────────────────────────────────────────
- hdf5 1.14.0 nompi_hb72d44e_103 conda-forge
+ hdf5 1.12.2 nompi_h4df4325_101 conda-forge/linux-64 Cached
- libdeflate 1.18 h0b41bf4_0 conda-forge
+ libdeflate 1.17 h0b41bf4_0 conda-forge/linux-64 Cached
- libjpeg-turbo 2.1.5.1 h0b41bf4_0 conda-forge
+ libjpeg-turbo 2.1.4 h166bdaf_0 conda-forge/linux-64 Cached
- libnetcdf 4.9.2 nompi_hf3f8848_103 conda-forge
+ libnetcdf 4.8.1 nompi_h261ec11_106 conda-forge/linux-64 Cached
- netcdf4 1.6.3 nompi_py310h2d0b64f_102 conda-forge
+ netcdf4 1.6.2 nompi_py310h55e1e36_100 conda-forge/linux-64 Cached
Summary:
Install: 1 packages
Change: 4 packages
Downgrade: 5 packages
```
However constraining libnetcdf, seems to create a new error with dask:
```
INFO:distributed.batched:Batched Comm Closed <TCP (closed) Scheduler connection to worker local=tcp://127.0.0.1:49048 remote=tcp://127.0.0.1:53328>
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix_error_test/lib/python3.10/site-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File "/home/sm_carni/.conda/envs/climix_error_test/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
value = future.result()
File "/home/sm_carni/.conda/envs/climix_error_test/lib/python3.10/site-packages/distributed/comm/tcp.py", line 269, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
```
can be solved by constraining `netCDF4==1.6.0` but I am not sure if this is the right approach.0.18 (Gull Olle)Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/299Add unit tests for index functions2023-04-21T15:56:13ZCarolina NilssonAdd unit tests for index functionsImplemented tests:
- [x] CountLevelCrossings
- [x] CountOccurrences
- [x] CountJointOccurrencesPrecipitationTemperature
- [x] CountJointOccurrencesTemperature
- [x] DiurnalTemperatureRange
- [x] ExtremeTemperatureRange
- [ ] FirstOccurre...Implemented tests:
- [x] CountLevelCrossings
- [x] CountOccurrences
- [x] CountJointOccurrencesPrecipitationTemperature
- [x] CountJointOccurrencesTemperature
- [x] DiurnalTemperatureRange
- [x] ExtremeTemperatureRange
- [ ] FirstOccurrence (moved to later release)
- [x] InterdayDiurnalTemperatureRange
- [ ] LastOccurrence (moved to later release)
- [x] Percentile
- [x] Statistics
- [x] ThresholdedPercentile
- [x] ThresholdedStatistics
- [x] RunningStatistics
- [x] ThresholdedRunningStatistics
- [x] TemperatureSum
The following unit tests needs to be reviewed from a scientific point of view:
* [x] CountLevelCrossings - Gustav
* [ ] CountOccurrences - Gustav
* [ ] CountJointOccurrencesPrecipitationTemperature
* [ ] CountJointOccurrencesTemperature
* [x] DiurnalTemperatureRange - Renate
* [x] ExtremeTemperatureRange - Renate
* [x] InterdayDiurnalTemperatureRange - Renate
* [ ] Statistics
* [x] ThresholdedStatistics - Renate
* [ ] RunningStatistics (complex output with post-processing included)
* [ ] ThresholdedRunningStatistics (complex output with post-processing included)
* [ ] TemperatureSum
E.g.,
```
TEST_COUNT_LEVEL_CROSSINGS_PARAMETERS = [
(
{"data": (-1) * np.arange(12).reshape(2, 2, 3), "units": "degree_Celsius"},
{"data": np.arange(12).reshape(2, 2, 3), "units": "degree_Celsius"},
{"data": 0, "units": "degree_Celsius", "standard_name": "air_temperature"},
np.array([[1, 2, 2], [2, 2, 2]]),
), # ordinary np
]
parameter_names = "f_cube_tasmin, f_cube_tasmax, f_first_threshold, expected"
```
The index function is count_level_crossings, bellow the parameter list with the data we can see the parameter names. Such that,
```
f_cube_tasmin = {"data": (-1) * np.arange(12).reshape(2, 2, 3), "units": "degree_Celsius"}
f_cube_tasmax = {"data": np.arange(12).reshape(2, 2, 3), "units": "degree_Celsius"}
f_first_threshold = {"data": 0, "units": "degree_Celsius", "standard_name": "air_temperature"}
expected = np.array([[1, 2, 2], [2, 2, 2]])
```0.18 (Gull Olle)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/298Update clix-meta files to v0.5.12023-03-14T09:52:37ZJoakim LöwUpdate clix-meta files to v0.5.1Replace `index_definitions.yml` and `variables.yml` with files released with clix-meta v0.5.1 and update Climix to handle it.Replace `index_definitions.yml` and `variables.yml` with files released with clix-meta v0.5.1 and update Climix to handle it.0.18 (Gull Olle)Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/287Produce a nice summary of which indices are available in climix2023-04-23T13:41:47ZLars BärringProduce a nice summary of which indices are available in climixUsers have requested to have access, possibly published via SMHI internal web pages to which indices that climix can produce.
A starting point may be the output from `climix -x list`:
``` bash
~ >climix -x list
INFO:root:Activating sen...Users have requested to have access, possibly published via SMHI internal web pages to which indices that climix can produce.
A starting point may be the output from `climix -x list`:
``` bash
~ >climix -x list
INFO:root:Activating sentry (automatic error reporting)
INFO:root:Looking for metadata in directory /home/sm_lbarr/CODE/climix/climix/etc
INFO:root:Looking for metadata in directory /etc/climix
INFO:root:Looking for metadata in directory /home/sm_lbarr/.config/climix
INFO:root:Reading index definitions from file /home/sm_lbarr/CODE/climix/climix/etc/metadata.yml
INFO:root:Reading index definitions from file /home/sm_lbarr/CODE/climix/climix/etc/variables.yml
INFO:root:Reading index definitions from file /home/sm_lbarr/CODE/climix/climix/etc/index_definitions.yml
Available indices are:
['fd', 'tnlt2', 'tnltm2', 'tnltm20', 'id', 'su', 'txge30', 'txge35', 'tr', 'tmge5', 'tmlt5', 'tmge10', 'tmlt10',
'tngt{TT}', 'tnlt{TT}', 'tnge{TT}', 'tnle{TT}', 'txgt{TT}', 'txlt{TT}', 'txge{TT}', 'txle{TT}', 'tmgt{TT}',
'tmlt{TT}', 'tmge{TT}', 'tmle{TT}', 'ctngt{TT}', 'cfd', 'csu', 'ctnlt{TT}', 'ctnge{TT}', 'ctnle{TT}', 'ctxgt{TT}',
'ctxlt{TT}', 'ctxge{TT}', 'ctxle{TT}', 'ctmgt{TT}', 'ctmlt{TT}', 'ctmge{TT}', 'ctmle{TT}', 'txx', 'tnx', 'txn', '
tnn', 'txm', 'tnm', 'tmx', 'tmn', 'tmm', 'txmax', 'tnmax', 'txmin', 'tnmin', 'txmean', 'tnmean', 'tmmax', 'tmmin',
'tmmean', 'tn10p', 'tx10p', 'tn90p', 'tx90p', 'tg10p', 'tg90p', 'txgt50p', 'txgt{PRC}p', 'tngt{PRC}p', 'tmgt{PRC}p',
'txlt{PRC}p', 'tnlt{PRC}p', 'tmlt{PRC}p', 'dtr', 'vdtr', 'etr', 'tx{PRC}pctl', 'tn{PRC}pctl', 'tm{PRC}pctl',
'hd17', 'hddheat{TT}', 'ddgt{TT}', 'cddcold{TT}', 'ddlt{TT}', 'gddgrow{TT}', 'gd4', 'r10mm', 'r20mm', 'r{RT}mm',
'wetdays', 'rr1', 'cdd', 'cwd', 'prcptot', 'sdii', 'r{PRC}pctl', 'r{PRC}pDAYS', 'rx1day', 'rx5day', 'rx{ND}day',
'rh', 'rr', 'pp', 'tg', 'tn', 'tx', 'sd', 'sd1', 'sd5cm', 'sd50cm', 'sd{D}cm', 'ss', 'fxx', 'fg6bft', 'fgcalm',
'fg', 'nzero', 'maxdtr']
```
This can of course be expanded in various directions, e.g. to
* list the indices per file that is scanned,
* also give the `long_name`, `OUTPUT_unit`, input variable(s), ...
* see #57 for another and much more complex idea0.18 (Gull Olle)Klaus ZimmermannKlaus Zimmermann