climix issueshttps://git.smhi.se/climix/climix/-/issues2023-03-13T09:32:26Zhttps://git.smhi.se/climix/climix/-/issues/256Consecutive index fails on 1 dimensional cubes2023-03-13T09:32:26ZErik HolmgrenConsecutive index fails on 1 dimensional cubesThe index `ctxge{TT}` fails on 1-dimensional cubes (i.e. cubes with only a time dimension, spatial averages etc.). Klaus did some digging and we found that the `spell_kernel` expects a 3-dimensional cube. A quick fix is to add dummy coor...The index `ctxge{TT}` fails on 1-dimensional cubes (i.e. cubes with only a time dimension, spatial averages etc.). Klaus did some digging and we found that the `spell_kernel` expects a 3-dimensional cube. A quick fix is to add dummy coordinates to such a cube before computing the index, but it would still be good if these types of indices worked on all cubes.
I haven't tried all the consecutive indices but I assume this issue is true for all of them. To get the true "exception" trying this, one has to disable numba in the `spell_kernel`.0.17 (Black Lion)Erik HolmgrenErik Holmgrenhttps://git.smhi.se/climix/climix/-/issues/254Climix fails when using threaded or single-threaded scheduler2022-09-15T12:27:03ZJoakim LöwClimix fails when using threaded or single-threaded schedulerRunning Climix with `-d threaded` or `-d single-threaded` causes a crash. The corresponding scheduler classes (LocalThreadsScheduler and SingleThreadedScheduler) seems to be missing an expected 'client' member.
$ climix -s -e -l err...Running Climix with `-d threaded` or `-d single-threaded` causes a crash. The corresponding scheduler classes (LocalThreadsScheduler and SingleThreadedScheduler) seems to be missing an expected 'client' member.
$ climix -s -e -l error -d threaded -x txx /home/rossby/prod/201137/netcdf/day/tasmax_EUR-11_ICHEC-EC-EARTH_historical_r12i1p1_SMHI-RCA4_v1_day_19700101-19701231.nc
Traceback (most recent call last):
File "/home/sm_joalo/.conda/envs/omni/bin/climix", line 33, in <module>
sys.exit(load_entry_point('climix', 'console_scripts', 'climix')())
File "/home/sm_joalo/dev/repos/climix/climix/main.py", line 179, in main
do_main(
File "/home/sm_joalo/dev/repos/climix/climix/main.py", line 158, in do_main
result = index(input_data, client=scheduler.client, sliced_mode=sliced_mode)
File "/home/sm_joalo/dev/repos/climix/climix/index.py", line 52, in __call__
aggregated = multicube_aggregated_by(
File "/home/sm_joalo/dev/repos/climix/climix/iris.py", line 148, in multicube_aggregated_by
aggregateby_cube = aggregator.post_process(
File "/home/sm_joalo/dev/repos/climix/climix/aggregators.py", line 65, in post_process
data = self.compute_pre_result(data, client, sliced_mode)
File "/home/sm_joalo/dev/repos/climix/climix/aggregators.py", line 42, in compute_pre_result
d = client.persist(d)
AttributeError: 'NoneType' object has no attribute 'persist'https://git.smhi.se/climix/climix/-/issues/253Newer version of Iris lazy-load coord data, causes crash in climix during save2022-09-14T09:42:42ZJoakim LöwNewer version of Iris lazy-load coord data, causes crash in climix during saveIn a newer version of Iris coord data of a cube is lazy-loaded. Therefore, when saving output in climix, the data may not exist in memory yet, which causes a crash. Solution: touch the coord data before saving.In a newer version of Iris coord data of a cube is lazy-loaded. Therefore, when saving output in climix, the data may not exist in memory yet, which causes a crash. Solution: touch the coord data before saving.Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/243Seasonal periods are broken2021-07-28T15:37:22ZKlaus ZimmermannSeasonal periods are brokenAt the moment, seasonal periods can only be calculated if a single season is specified with, e.g., `seasonal: JASOND` in the index definition.
Instead, it should be possible to calculate seasonal indices for the standard seasons without ...At the moment, seasonal periods can only be calculated if a single season is specified with, e.g., `seasonal: JASOND` in the index definition.
Instead, it should be possible to calculate seasonal indices for the standard seasons without any specification.
This issue is more narrow than #227 and only intents to get the current, limited functionality working again prior to a more comprehensive overhaul.0.14Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/242Index function SpellLength is broken2021-07-27T13:17:58ZKlaus ZimmermannIndex function SpellLength is brokenThe issue is a technical one: The computation of the `meta` information for a dask routine is not working due to an incorrect argument to `np.array`.
Fortunately, the fix is straightforward.The issue is a technical one: The computation of the `meta` information for a dask routine is not working due to an incorrect argument to `np.array`.
Fortunately, the fix is straightforward.0.14Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/230Remove empty/inon-existing directories from input file list2021-07-06T15:10:40ZLars BärringRemove empty/inon-existing directories from input file listIf brace-expansion is used for creating the input filelist it may happen that empty/nonexisting directories are included in the list of filenames. Such directories/filenames, that have to be removed, will include the globbing character "*".If brace-expansion is used for creating the input filelist it may happen that empty/nonexisting directories are included in the list of filenames. Such directories/filenames, that have to be removed, will include the globbing character "*".Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/226wetdays djf error on the maxium number of days2021-07-22T14:40:57ZRamon Fuentes Francowetdays djf error on the maxium number of daysWhen using a modified version of the wetdays for djf, the maximum number of days that comes out in the output file is 30, I suspect it is only summing in one month, and not on the entire djf season.
See below the modified version I used...When using a modified version of the wetdays for djf, the maximum number of days that comes out in the output file is 30, I suspect it is only summing in one month, and not on the entire djf season.
See below the modified version I used:
```yaml
wetdays_djf:
reference: CLIPC
period:
allowed:
annual:
seasonal: 'djf'
monthly:
default: seasonal
output:
var_name: "wetdays"
standard_name: number_of_days_with_lwe_thickness_of_precipitation_amount_above_threshold
proposed_standard_name: number_of_occurrences_with_lwe_thickness_of_precipitation_amount_at_or_above_threshold
long_name: "Number of Wet Days (precip >= 1 mm)"
units: "days"
cell_methods:
- time: sum within days
- time: sum over days
input:
data: pr
index_function:
name: count_occurrences
parameters:
threshold:
kind: quantity
standard_name: lwe_precipitation_rate
long_name: "Wet day threshold"
data: 1
units: "mm day-1"
condition:
kind: operator
operator: ">"
ET:
short_name:
long_name:
definition:
comment:
```0.14https://git.smhi.se/climix/climix/-/issues/224CountPercentileOccurrences doesn't respect the condition operator2021-05-04T11:56:15ZKlaus ZimmermannCountPercentileOccurrences doesn't respect the condition operatorOne of the issues that surfaced in #205 is that `CountPercentileOccurrences` doesn't respect the condition operator, i.e. it always checks that the data is less than the threshold, regardless of the operator given in the index definition...One of the issues that surfaced in #205 is that `CountPercentileOccurrences` doesn't respect the condition operator, i.e. it always checks that the data is less than the threshold, regardless of the operator given in the index definition. This should be corrected.0.13.2Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/220Pin iris also in environment.yml2021-04-14T15:10:01ZKlaus ZimmermannPin iris also in environment.ymlThe pinning of iris (see #218) should happen also in the `environment.yml` file.The pinning of iris (see #218) should happen also in the `environment.yml` file.0.13.2Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/218Pin iris version2021-04-14T15:07:43ZKlaus ZimmermannPin iris versionThe current code is not compatible with iris 3, so we should pin to iris<3 to guarantee a working code.The current code is not compatible with iris 3, so we should pin to iris<3 to guarantee a working code.0.13.2Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/215Climix does not gracefully handle single node / singe core2022-09-15T12:18:50ZLars BärringClimix does not gracefully handle single node / singe coreClimix crashes when running on a shared node with `interactive -N 1 -n 1`:
```
>climix -s -e -x txx /home/rossby/prod/201137/netcdf/day/tasmax_EUR-11_ICHEC-EC-EARTH_historical_r12i1p1_SMHI-RCA4_v1_day_*.nc
INFO:root:Loading metadata
T...Climix crashes when running on a shared node with `interactive -N 1 -n 1`:
```
>climix -s -e -x txx /home/rossby/prod/201137/netcdf/day/tasmax_EUR-11_ICHEC-EC-EARTH_historical_r12i1p1_SMHI-RCA4_v1_day_*.nc
INFO:root:Loading metadata
Traceback (most recent call last):
File "/home/sm_lbarr/.conda/envs/climix-devel-3/bin/climix", line 11, in <module>
load_entry_point('climix', 'console_scripts', 'climix')()
File "/home/sm_lbarr/CODE/climix/climix/main.py", line 146, in main
with setup_scheduler(args) as scheduler:
File "/home/sm_lbarr/CODE/climix/climix/dask_setup.py", line 160, in setup_scheduler
return scheduler(**scheduler_kwargs)
File "/home/sm_lbarr/CODE/climix/climix/dask_setup.py", line 68, in __init__
memory_limit = (system.MEMORY_LIMIT*.9) / n_workers
ZeroDivisionError: float division by zero
```
Preferably it should instead give a graceful error message.0.16Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/211Fix output units for index function count_occurrences2022-11-16T08:00:57ZLars BärringFix output units for index function count_occurrencesIt seems that the index function `count_occurrences` expect output to have units `day`. This is however the `proposed_unit`, which not consistent with current the current canonical unit `1` as defined by the `standard names`. This has pr...It seems that the index function `count_occurrences` expect output to have units `day`. This is however the `proposed_unit`, which not consistent with current the current canonical unit `1` as defined by the `standard names`. This has probably remained under the radar because I was until recently using a really old version of `index_definition.yml`. If simple to fix then target bugfix milestone 13.2, else postpone it to a later milestone.0.15Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/210Change type of _FillValue2021-02-16T17:10:12ZLars BärringChange type of _FillValueThe `cfchecker` consistently reports the following message:
```
CHECKING NetCDF FILE: cdd_NORDIC-3_SMHI-UERRA-Harmonie_RegRean_v1_Gridpp_v0.9_day_yr_19601231-20181230.nc
=====================
Using CF Checker Version 4.0.0
Checking again...The `cfchecker` consistently reports the following message:
```
CHECKING NetCDF FILE: cdd_NORDIC-3_SMHI-UERRA-Harmonie_RegRean_v1_Gridpp_v0.9_day_yr_19601231-20181230.nc
=====================
Using CF Checker Version 4.0.0
Checking against CF Version CF-1.7
Using Standard Name Table Version 77 (2021-01-19T13:38:50Z)
Using Area Type Table Version 10 (23 June 2020)
Using Standardized Region Name Table Version 4 (18 December 2018)
------------------
Checking variable: cdd
------------------
INFO: Invalid Type for attribute: _FillValue <class 'numpy.float32'>
```
While not reported as an `ERROR` it seems like a simple thing to change this to a normal `float32`0.13.2Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/208index DTR crashes2021-04-19T20:26:08ZLars Bärringindex DTR crashesIt seems that climix now crashes for multiple input streams, example `dtr`. Here are two test cases both using the master branch that is updated wrt repo:
**With one file each for tasmax and tasmin:**
```python
(climix-devel-3) ~/PROJ/...It seems that climix now crashes for multiple input streams, example `dtr`. Here are two test cases both using the master branch that is updated wrt repo:
**With one file each for tasmax and tasmin:**
```python
(climix-devel-3) ~/PROJ/test_climix >climix -s -e -x dtr /home/sm_lbarr/PROJ/test_climix/indata/CLIMPACT2_TEST/tmin_AUS_climpact2-sampledata_gridded-obs_v1_ET-SCI_v1_1991-2010.nc /home/sm_lbarr/PROJ/test_climix/indata/CLIMPACT2_TEST/tmax_AUS_climpact2-sampledata_gridded-obs_v1_ET-SCI_v1_1991-2010.nc
INFO:root:Loading metadata
INFO:root:Scheduler ready; starting main program.
INFO:root:Preparing indices
INFO:root:Found implementation for index_function <diurnal_temperature_range> from distribution <climix 0.13.1>
INFO:root:Starting calculations for index <climix.index.Index object at 0x2b8f67a6af70>
INFO:root:Building output filename
INFO:root:Preparing input data
INFO:root:Calculating index
INFO:root:Starting preprocess
INFO:root:Finished preprocess
INFO:root:Data found for input low_data
INFO:root:Data found for input high_data
INFO:root:Adding coord categorisation.
INFO:root:Preparing cubes
INFO:root:Setting up aggregation
INFO:root:Calculation took 16.5291 seconds.
Traceback (most recent call last):
File "/home/sm_lbarr/.conda/envs/climix-devel-3/bin/climix", line 11, in <module>
load_entry_point('climix', 'console_scripts', 'climix')()
File "/home/sm_lbarr/CODE/climix/climix/main.py", line 150, in main
do_main(index_catalog, args.indices, args.datafiles,
File "/home/sm_lbarr/CODE/climix/climix/main.py", line 128, in do_main
result = index(input_data,
File "/home/sm_lbarr/CODE/climix/climix/index.py", line 46, in __call__
aggregated = multicube_aggregated_by(cube_mapping, coord_name,
File "/home/sm_lbarr/CODE/climix/climix/iris.py", line 124, in multicube_aggregated_by
result = list(map(agg, groupby_subcubes))
File "/home/sm_lbarr/CODE/climix/climix/iris.py", line 120, in agg
data = data_getter(cube)
File "/home/sm_lbarr/CODE/climix/climix/iris.py", line 90, in data_getter
return cube.lazy_data()
AttributeError: 'dict' object has no attribute 'lazy_data'
```
**With multiple input files for each variable:**
```python
climix -e -s -x dtr /home/rossby/imports/cordex/EUR-11/CLMcom-CCLM4-8-17/v1/ICHEC-EC-EARTH/r12i1p1/rcp85/bc/links-hist-scn/day/tasmax_EUR-11_ICHEC-EC-EARTH_rcp85_r12i1p1_CLMcom-CCLM4-8-17_v1_day_*.nc /home/rossby/imports/cordex/EUR-11/CLMcom-CCLM4-8-17/v1/ICHEC-EC-EARTH/r12i1p1/rcp85/bc/links-hist-scn/day/tasmin_EUR-11_ICHEC-EC-EARTH_rcp85_r12i1p1_CLMcom-CCLM4-8-17_v1_day_*.nc
INFO:root:Loading metadata
INFO:root:Scheduler ready; starting main program.
INFO:root:Preparing indices
INFO:root:Calculation took 0.0005 seconds.
Traceback (most recent call last):
File "/home/sm_lbarr/.conda/envs/climix-devel-3/bin/climix", line 11, in <module>
load_entry_point('climix', 'console_scripts', 'climix')()
File "/home/sm_lbarr/CODE/climix/climix/main.py", line 150, in main
do_main(index_catalog, args.indices, args.datafiles,
File "/home/sm_lbarr/CODE/climix/climix/main.py", line 118, in do_main
indices = index_catalog.prepare_indices(requested_indices)
File "/home/sm_lbarr/CODE/climix/climix/metadata.py", line 335, in prepare_indices
definition = self.get_index_definition(index)
File "/home/sm_lbarr/CODE/climix/climix/metadata.py", line 300, in get_index_definition
return self.indices[index]
KeyError: '/home/rossby/imports/cordex/EUR-11/CLMcom-CCLM4-8-17/v1/ICHEC-EC-EARTH/r12i1p1/rcp85/bc/links-hist-scn/day/tasmax_EUR-11_ICHEC-EC-EARTH_rcp85_r12i1p1_CLMcom-CCLM4-8-17_v1_day_19510101-19551231.nc'
```0.13.2Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/206Editor bug fix: change OUTPUT_user_units to OUTPUT_proposed_units2020-10-26T18:16:19ZLars BärringEditor bug fix: change OUTPUT_user_units to OUTPUT_proposed_unitsLars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/205correction of index_function count_percentile_occurrences (tn10p and similar)2021-05-04T11:54:59ZLars Bärringcorrection of index_function count_percentile_occurrences (tn10p and similar)Test runs show that `tn10p` is OK with only small deviations wrt reference data. These devitions may be related to the problems listed below.
* [x] *Trivial:* In setup.py move index function `count_percentile_occurrences` to its place i...Test runs show that `tn10p` is OK with only small deviations wrt reference data. These devitions may be related to the problems listed below.
* [x] *Trivial:* In setup.py move index function `count_percentile_occurrences` to its place in alphabetical order.
* [x] `tn90p`, i.e. *high percentile* and `>` gives an early error cf. `~/PROJ/test_climix >more slurm-12739678.out`
* [x] `txgt05p`, i.e. *low percentile* and `>` produces the output file, but the data is not correct, cf. `/nobackup/rossby20/sm_lbarr/PROJ/test_climix/txgt5p_EUR-11_ICHEC-EC-EARTH_rcp85_r12i1p1_CLMcom-CCLM4-8-17_v1_day_yr_19510101-21001231.nc`. And there is a suspicious error message at the end of the log file, cf. `~/PROJ/test_climix >cat slurm-12742194.out`
* [ ] ~~In case of one percentile value as index function parameter; store it as an auxiliary coordinate. (How to handle the future possibility of several percentiles?).~~0.13.2Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/202Rename index function percentile_occurrence for consistency2020-10-21T14:13:00ZLars BärringRename index function percentile_occurrence for consistencyIn `setup.py` we now (tag 0.13) have the line
`'percentile_occurrence=' 'climix.index_functions:PercentileOccurrence',`
However, as the index function really **counts** the number of occurrence**s** the name should be changed to reflec...In `setup.py` we now (tag 0.13) have the line
`'percentile_occurrence=' 'climix.index_functions:PercentileOccurrence',`
However, as the index function really **counts** the number of occurrence**s** the name should be changed to reflect this. Such a change is consistent with existing names. Therefor the index function name should be changed to:
`'count_percentile_occurrences=' 'climix.index_functions:CountPercentileOccurrences',`0.13.1Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/184SCALAR_REDUCERS is missing mean2020-04-22T09:28:25ZKlaus ZimmermannSCALAR_REDUCERS is missing meanThe `SCALAR_REDUCERS` in `climix.index_functions.support` are missing an implementation for `mean`.
This triggers an exception in the ReducerMixin and should be added.The `SCALAR_REDUCERS` in `climix.index_functions.support` are missing an implementation for `mean`.
This triggers an exception in the ReducerMixin and should be added.0.13Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/167Do not "format" standard names2020-02-19T15:46:00ZLars BärringDo not "format" standard namesSome metadata elements can be "parameterised" to include the [numeric] value instead of a placeholder within braces (e.g. `{PRC}`). However this should not be done for either `standard_name` or `proposed_standard_name`, irrespective of w...Some metadata elements can be "parameterised" to include the [numeric] value instead of a placeholder within braces (e.g. `{PRC}`). However this should not be done for either `standard_name` or `proposed_standard_name`, irrespective of whether it is the index variable or and auxiliary coordinate.
Cf. disussion in !115 for background.0.12.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/136Bug in threshold data type conversion2019-11-21T13:02:45ZLars BärringBug in threshold data type conversionFor all tested precip indices climix crashes, see below. Probably the threshold should be casted as `np.float32` already when read into climix?
```
climix -s -e -x cdd /home/sm_lbarr/PROJ/test_climix/indata/CLIMPACT2_TEST/precip_AUS_...For all tested precip indices climix crashes, see below. Probably the threshold should be casted as `np.float32` already when read into climix?
```
climix -s -e -x cdd /home/sm_lbarr/PROJ/test_climix/indata/CLIMPACT2_TEST/precip_AUS_climpact2-sampledata_gridded-obs_v1_ET-SCI_v1_1991-2010.nc
INFO:root:Loading metadata
INFO:root:Scheduler ready; starting main program.
INFO:root:Preparing indices
INFO:root:Found implementation for index_function <spell_length> from distribution <climix 0.9.0+14.g29e95e2>
INFO:root:Starting calculations for index <climix.index.Index object at 0x2ab6840234a8>
INFO:root:Building output filename
INFO:root:Preparing input data
INFO:root:Calculating index
INFO:root:Adding coord categorisation.
INFO:root:Extracting period cube
INFO:root:Preparing cube
INFO:root:Calculation took 3.6670 seconds.
Traceback (most recent call last):
File "/home/sm_lbarr/.conda/envs/climix-devel-3/bin/climix", line 11, in <module>
load_entry_point('climix', 'console_scripts', 'climix')()
File "/home/sm_lbarr/CODE/climix/climix/main.py", line 126, in main
args.output_template, args.sliced_mode)
File "/home/sm_lbarr/CODE/climix/climix/main.py", line 105, in do_main
result = index(input_data)
File "/home/sm_lbarr/CODE/climix/climix/index.py", line 23, in __call__
self.index_function.prepare(sub_cube)
File "/home/sm_lbarr/CODE/climix/climix/index_functions.py", line 200, in prepare
input_cube.standard_name)
File "/home/sm_lbarr/CODE/climix/climix/util/__init__.py", line 9, in change_units
change_pr_units(cube_or_coord, new_units, new_standard_name)
File "/home/sm_lbarr/CODE/climix/climix/util/change_pr_units.py", line 118, in change_pr_units
cube_or_coord.points *= float(conv_factor)
numpy.core._exceptions.UFuncTypeError: Cannot cast ufunc 'multiply' output from dtype('float64') to dtype('int64') with casting rule 'same_kind'
```0.10.0Klaus ZimmermannKlaus Zimmermann