climix issueshttps://git.smhi.se/climix/climix/-/issues2024-03-22T08:20:19Zhttps://git.smhi.se/climix/climix/-/issues/351Enhancing index cdd with accurate time stamps/vector [0/2]2024-03-22T08:20:19ZRenate WilckeEnhancing index cdd with accurate time stamps/vector [0/2]Cdd (consecutive dry days) and other consecutive indicators give the maximum spell length of a condition. Right now the time coordinate only contains the middle value of the calculated period (annual, monthly, ...). It would be a great a...Cdd (consecutive dry days) and other consecutive indicators give the maximum spell length of a condition. Right now the time coordinate only contains the middle value of the calculated period (annual, monthly, ...). It would be a great advantage for analysis to know WHEN the spell happens and to put that start date in the time coordinate instead.
E.g. if I calculate cdd annually, and the spell starts on the May the 4th, it would be good to have 2023_05_04 in the time coordinate instead of 2023_07_02 (the middle of the year).https://git.smhi.se/climix/climix/-/issues/364Investigate how and where in code to add time stamp2024-03-05T14:52:39ZJoakim LöwInvestigate how and where in code to add time stampSee discussion in #351See discussion in #351https://git.smhi.se/climix/climix/-/issues/365Implement time stamp enhancement2024-03-05T14:53:03ZJoakim LöwImplement time stamp enhancementSee discussion in #351See discussion in #351https://git.smhi.se/climix/climix/-/issues/353New clix-meta definitions [0/2]2024-02-20T12:17:44ZJoakim LöwNew clix-meta definitions [0/2]All indices considered internal should be moved from clix-meta. Afterwards, files in this repo related should be updated.All indices considered internal should be moved from clix-meta. Afterwards, files in this repo related should be updated.https://git.smhi.se/climix/climix/-/issues/354Update index definition files after clix-meta update2024-02-20T12:15:21ZJoakim LöwUpdate index definition files after clix-meta updatehttps://git.smhi.se/climix/climix/-/issues/355Update index docs with added and removed indices2024-02-20T12:14:58ZJoakim LöwUpdate index docs with added and removed indiceshttps://git.smhi.se/climix/climix/-/issues/356Update dependencies [1/3]2024-03-25T08:35:05ZJoakim LöwUpdate dependencies [1/3]https://git.smhi.se/climix/climix/-/issues/362Systems tasks [1/2]2024-03-25T08:34:41ZJoakim LöwSystems tasks [1/2]https://git.smhi.se/climix/climix/-/issues/357Remove pandas dependency2024-02-29T14:56:00ZJoakim LöwRemove pandas dependencyRemove pandas as dependency from
- environment.yml
- pyproject.toml
and remove from docs.Remove pandas as dependency from
- environment.yml
- pyproject.toml
and remove from docs.https://git.smhi.se/climix/climix/-/issues/361Sync dependencies lists2024-03-01T12:21:49ZJoakim LöwSync dependencies listsClimix has dependency lists in several places:
- climix repo: `environments.yml`
- climix repo: `pyproject.toml`
- conda-forge climix-feedstock: `recipe/meta.yaml`
They are not 100% in sync. Make sure to sync them just before next relea...Climix has dependency lists in several places:
- climix repo: `environments.yml`
- climix repo: `pyproject.toml`
- conda-forge climix-feedstock: `recipe/meta.yaml`
They are not 100% in sync. Make sure to sync them just before next release (0.22).https://git.smhi.se/climix/climix/-/issues/359Investigate RuntimeWarning2024-03-01T12:30:24ZCarolina NilssonInvestigate RuntimeWarningRunning `climix -l debug -o /nobackup/rossby27/users/sm_carni/data/tmp/output_files/out_climix_patch.nc -x fd /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20060...Running `climix -l debug -o /nobackup/rossby27/users/sm_carni/data/tmp/output_files/out_climix_patch.nc -x fd /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20060101-20101231.nc /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20110101-20151231.nc`
with climix installed causes RuntimeWarnings in numpy. We should find out why these warnings appear and if they have any effect on the run.https://git.smhi.se/climix/climix/-/issues/358Wrong version number on readthedocs2024-03-01T09:47:12ZJoakim LöwWrong version number on readthedocsThe latest release of climix on pypi is 0.21.1, but the documentation on readthedocs shows 0.22.The latest release of climix on pypi is 0.21.1, but the documentation on readthedocs shows 0.22.https://git.smhi.se/climix/climix/-/issues/368Add custom load callback and constraint2024-03-18T13:38:01ZJoakim LöwAdd custom load callback and constraintWhen loading an iris cube, a callback and constraint can be specified. Add possibility to add such callback and constraint from command line, e.g. pointing to a separate python file, or to specify the callback and constraint when using t...When loading an iris cube, a callback and constraint can be specified. Add possibility to add such callback and constraint from command line, e.g. pointing to a separate python file, or to specify the callback and constraint when using the climix api.https://git.smhi.se/climix/climix/-/issues/363Investigate ruff as linter and formatter2024-03-01T13:25:39ZJoakim LöwInvestigate ruff as linter and formatterhttps://git.smhi.se/climix/climix/-/issues/288SMHI start of seasons2024-02-02T12:08:17ZLars BärringSMHI start of seasonsImplement indicators (indices) for SMHI definitions of [start of] seasons. Possibly this can be done using existing functionality (`gsstart`?) and only new entries in the yaml file.
First step is to find out exactly how the season trans...Implement indicators (indices) for SMHI definitions of [start of] seasons. Possibly this can be done using existing functionality (`gsstart`?) and only new entries in the yaml file.
First step is to find out exactly how the season transitions are defined.0.22Gustav StrandbergGustav Strandberghttps://git.smhi.se/climix/climix/-/issues/285Implementation of index function for WSDI, CSDI2023-10-26T13:13:52ZLars BärringImplementation of index function for WSDI, CSDIWSDI and CSDI are (briefly) defined by ETCCDI and ET-SCI as
|VarName | OUTPUT_long_name |
|----------|------------------------------...WSDI and CSDI are (briefly) defined by ETCCDI and ET-SCI as
|VarName | OUTPUT_long_name |
|----------|-----------------------------------------------------------------------------------------------------------------------|
| wsdi | Warm Spell Duration Index, count of days with at least 6 consecutive days when Tmax > 90th percentile |
| wsdi{ND} | User-defined Warm Spell Duration Index, count of days with at least {ND} consecutive days when Tmax > 90th percentile |
| csdi | Cold Spell Duration Index, count of days with at least 6 consecutive days when Tmin < 10th percentile |
| csdi{ND} | User-defined Cold Spell Duration Index, count of days with at least # consecutive days when Tmin < 10th percentile |https://git.smhi.se/climix/climix/-/issues/286Index function for CW, CD, WW, WD2023-02-27T20:14:19ZLars BärringIndex function for CW, CD, WW, WDECA&D defines these two-variable indices as:
| VarName | OUTPUT_long_name |
|---------|---------------------------------------...ECA&D defines these two-variable indices as:
| VarName | OUTPUT_long_name |
|---------|--------------------------------------------------------------------------------------------------------------------------------|
| CD | Days with TG < 25th percentile of daily mean temperature and RR < 25th percentile of daily precipitation sum ("cold/dry days") |
| CW | Days with TG < 25th percentile of daily mean temperature and RR > 75th percentile of daily precipitation sum ("cold/wet days") |
| WD | Days with TG > 75th percentile of daily mean temperature and RR < 25th percentile of daily precipitation sum ("warm/dry days") |
| WW | Days with TG > 75th percentile of daily mean temperature and RR > 75th percentile of daily precipitation sum ("warm/wet days") |
where TG is "air **T**emperature near the **G**round ", i.e. the usual daily mean temperature `tas`, and RR is daily total precipitation `pr`.https://git.smhi.se/climix/climix/-/issues/57Generate command line help from index definitions2024-02-02T14:17:04ZKlaus ZimmermannGenerate command line help from index definitions0.22https://git.smhi.se/climix/climix/-/issues/53Add yaml schema2023-03-31T14:03:40ZKlaus ZimmermannAdd yaml schemaHaving a schema to validate the yaml file against would guarantee syntactical consistency.Having a schema to validate the yaml file against would guarantee syntactical consistency.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/118Multiple thresholds2023-05-03T13:00:27ZLars BärringMultiple thresholdsMany indices are based on a threshold. For many of these indices it is relevant to have several thresholds E.g. for Summer Days the threshold is tasmax > 25 degC. It is often requested to have alternative thresholds e.g. 21, 23, 25, 27, ...Many indices are based on a threshold. For many of these indices it is relevant to have several thresholds E.g. for Summer Days the threshold is tasmax > 25 degC. It is often requested to have alternative thresholds e.g. 21, 23, 25, 27, 29 degC.
The index name should then be txgtn (TBC).
Cf. #73https://git.smhi.se/climix/climix/-/issues/150Implement Extreme Value Statistics2023-10-25T11:44:17ZLars BärringImplement Extreme Value StatisticsImplement basic Extreme Value statistics. Could use https://git.smhi.se/a001257/returnvalue/tree/classes as starting point. This code is still very much in development.Implement basic Extreme Value statistics. Could use https://git.smhi.se/a001257/returnvalue/tree/classes as starting point. This code is still very much in development.https://git.smhi.se/climix/climix/-/issues/156Refactor metadata dataclasses to contain constructors as classmethods2023-03-31T14:03:31ZKlaus ZimmermannRefactor metadata dataclasses to contain constructors as classmethodsThere are a number of factory-methods for dataclasses in metadata.py. It would be good to move them to classmethods in the respective dataclasses.There are a number of factory-methods for dataclasses in metadata.py. It would be good to move them to classmethods in the respective dataclasses.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/176advanced drought related indices2023-04-27T07:43:55ZLars Bärringadvanced drought related indicesThe indices SPI and SPEI are included in the ET-SCI list, but has so far been beyond the horizon. However I just saw a [python implementation](https://github.com/monocongo/climate_indices) that seems to be solid enough in terms of instit...The indices SPI and SPEI are included in the ET-SCI list, but has so far been beyond the horizon. However I just saw a [python implementation](https://github.com/monocongo/climate_indices) that seems to be solid enough in terms of institutional support and interesting for their approach towards large data efficiency. @a002160 have a look at some point -- something we could benefit from hooking up to? The other indices on their list are well established and widely used, but typically not considered as "indices in sense where we are coming from".Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/177Counting exceedance above/below threshold across differernt datasets.2023-02-13T15:14:03ZLars BärringCounting exceedance above/below threshold across differernt datasets.We have numerous indices counting exceedance above/below a fixed threshold. There is need to be able to carry out such calculations across different datasets that may have different reference climate (i.e. there are "biases"...). In a si...We have numerous indices counting exceedance above/below a fixed threshold. There is need to be able to carry out such calculations across different datasets that may have different reference climate (i.e. there are "biases"...). In a simplistic way, and without invoking bias adjustment of the input data, this could be done by carrying out a quantile mapping of the threshold applied to the reference data. For this we new a couple of new index functions
We need a couple of new functions:
1. index function to calculate a percentile rank based on a fixed threshold, e.g. `tx{TT}rank` etc. This have to be done over some reference period, typically a 30 year period like 1971-2000.
2. index function to calculate a percentile value based on **spatially varying** percentiles (e.g. produced by 1.). This have to be done over a reference period, which typically is the same as used in 1.
3. index function to count the number of exceedances (above/below) a **spatially varying** threshold (e.g. produced by 2.)
In step 2 it should be possible to read the percentile array from the file produced by 1.
Step 2 and 3 could be combined. Irrespective of this, the result of 2 should be an auxiliary coordinate in step 3.https://git.smhi.se/climix/climix/-/issues/179tasmidpoint as standin for tas2023-04-04T21:15:34ZLars Bärringtasmidpoint as standin for tasThe well established indices were developed within the observational data communities. And in that context daily mean temperature, as in `tas: cell_method = "time: mean"`, brings about a lot of complications (given the limited and shifti...The well established indices were developed within the observational data communities. And in that context daily mean temperature, as in `tas: cell_method = "time: mean"`, brings about a lot of complications (given the limited and shifting observation times). Hence they settled for the draconian solution that **daily mean temperature is defined as `(TN+TX)/2`**, which in terms of CF would be `tas: cell_method = "time: midpoint` (assuming that the time bound specify daily data!), where *`tas`* should have another name, such as `tasmidpoint`.
We should have the possibility to compute `tas`-based indices on `tasmidpoint`, which then should be calculated on the fly as if input data had been specified as `low_data: tasmin, high_data: tasmax`https://git.smhi.se/climix/climix/-/issues/292Add linting test stage using GitLab CI /Kubernetes2024-02-02T12:06:13ZJoakim LöwAdd linting test stage using GitLab CI /KubernetesFirst step utilizing Kubernetes for CI: add a linting test stage to run linting on a MR.First step utilizing Kubernetes for CI: add a linting test stage to run linting on a MR.Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/141Backlog2024-02-02T10:46:11ZKlaus ZimmermannBacklogThere are a few things that we went with because they made sense at the time that I want to double check again.
This issue is a place to collect them.
- `normalize_axis` in index functions. Is it really necessary to accommodate negative...There are a few things that we went with because they made sense at the time that I want to double check again.
This issue is a place to collect them.
- `normalize_axis` in index functions. Is it really necessary to accommodate negative axis arguments and is this the way to do it?
- Do we need this strong separation of dask and numpy implementations? After some changes (perhaps both in numpy and dask) it seems that now even numpy functions will generally return dask arrays if presented with such.https://git.smhi.se/climix/climix/-/issues/329Index: Accumulated precip with Temperature below 02024-02-02T12:13:47ZJoakim LöwIndex: Accumulated precip with Temperature below 0Similar to index function: `TemperatureSum` (index: hd17)
This issue replaces #296
Note: check if it's ok to add index definition in `SMHI_extra.yml`.Similar to index function: `TemperatureSum` (index: hd17)
This issue replaces #296
Note: check if it's ok to add index definition in `SMHI_extra.yml`.Renate WilckeRenate Wilckehttps://git.smhi.se/climix/climix/-/issues/192index function to calculate annual range of monthly means2023-02-13T15:14:38ZLars Bärringindex function to calculate annual range of monthly meansNew index function to carry out the following:
* from daily data calculate the monthly means for all months
* based on the monthly means calculate the annual span
Use case: for temperature this functionality would form the basis for c...New index function to carry out the following:
* from daily data calculate the monthly means for all months
* based on the monthly means calculate the annual span
Use case: for temperature this functionality would form the basis for calculating various continentality indiceshttps://git.smhi.se/climix/climix/-/issues/198Other kinds of parameters2023-06-14T11:09:25ZKlaus ZimmermannOther kinds of parametersRight now, we have three kinds of parameters: Quantities, operators, and reducers.
For some indices, we need parameters such as reference periods or maybe others.
To allow for the passing of these, different options are possible:
- Int...Right now, we have three kinds of parameters: Quantities, operators, and reducers.
For some indices, we need parameters such as reference periods or maybe others.
To allow for the passing of these, different options are possible:
- Introduce new kinds as needed (increasing code complexity, but making the index specification type-safer)
- Encode everything in the current kinds (reference period would be a quantity)
- Introduce some form of lightweight kind, say `kind: scalar` that supports simple data with no units. For this case reference period would probably be a string following the [cmip6 convention for time_range (see page 14f)](http://goo.gl/v1drZl)
@a001257, what do you think?https://git.smhi.se/climix/climix/-/issues/221Calculation of number of days exceeding a percentile from another experiment2023-02-13T15:15:15ZRamon Fuentes FrancoCalculation of number of days exceeding a percentile from another experimentIt would be great to have an index in which it is calculated the number of days exceeding a percentile (annually, seasonally and monthly), for example, the number of days in the experiment ssp585 exceeding P95 of precipitation in the his...It would be great to have an index in which it is calculated the number of days exceeding a percentile (annually, seasonally and monthly), for example, the number of days in the experiment ssp585 exceeding P95 of precipitation in the historical experiment. This index could be applied for the following variables: pr, tasmax, tasmin, tas.https://git.smhi.se/climix/climix/-/issues/222Add `__repr__` methods to all classes2024-02-02T10:43:52ZKlaus ZimmermannAdd `__repr__` methods to all classeshttps://git.smhi.se/climix/climix/-/issues/251time.coord check for meaningful amount of time steps2023-06-09T14:04:32ZRenate Wilcketime.coord check for meaningful amount of time stepsclimix calculates an index also for a "year" with only one time step, resulting in a wrong value. That happens when the input data (daily fields) have been created from hourly data but the last time step is the first (00:00:00) of the ne...climix calculates an index also for a "year" with only one time step, resulting in a wrong value. That happens when the input data (daily fields) have been created from hourly data but the last time step is the first (00:00:00) of the next year.
It would be good to have a check for something like number of years mus result in at least number of days to avoid index created from one hour or day.https://git.smhi.se/climix/climix/-/issues/254Climix fails when using threaded or single-threaded scheduler2022-09-15T12:27:03ZJoakim LöwClimix fails when using threaded or single-threaded schedulerRunning Climix with `-d threaded` or `-d single-threaded` causes a crash. The corresponding scheduler classes (LocalThreadsScheduler and SingleThreadedScheduler) seems to be missing an expected 'client' member.
$ climix -s -e -l err...Running Climix with `-d threaded` or `-d single-threaded` causes a crash. The corresponding scheduler classes (LocalThreadsScheduler and SingleThreadedScheduler) seems to be missing an expected 'client' member.
$ climix -s -e -l error -d threaded -x txx /home/rossby/prod/201137/netcdf/day/tasmax_EUR-11_ICHEC-EC-EARTH_historical_r12i1p1_SMHI-RCA4_v1_day_19700101-19701231.nc
Traceback (most recent call last):
File "/home/sm_joalo/.conda/envs/omni/bin/climix", line 33, in <module>
sys.exit(load_entry_point('climix', 'console_scripts', 'climix')())
File "/home/sm_joalo/dev/repos/climix/climix/main.py", line 179, in main
do_main(
File "/home/sm_joalo/dev/repos/climix/climix/main.py", line 158, in do_main
result = index(input_data, client=scheduler.client, sliced_mode=sliced_mode)
File "/home/sm_joalo/dev/repos/climix/climix/index.py", line 52, in __call__
aggregated = multicube_aggregated_by(
File "/home/sm_joalo/dev/repos/climix/climix/iris.py", line 148, in multicube_aggregated_by
aggregateby_cube = aggregator.post_process(
File "/home/sm_joalo/dev/repos/climix/climix/aggregators.py", line 65, in post_process
data = self.compute_pre_result(data, client, sliced_mode)
File "/home/sm_joalo/dev/repos/climix/climix/aggregators.py", line 42, in compute_pre_result
d = client.persist(d)
AttributeError: 'NoneType' object has no attribute 'persist'https://git.smhi.se/climix/climix/-/issues/270Add a warning and raise a error if datafiles from unrelated datasets are used...2023-05-29T09:09:58ZCarolina NilssonAdd a warning and raise a error if datafiles from unrelated datasets are used together, e.g. tasmin and TNWhen running Climix with two input files from different datasets that uses two different variable names, e.g., tasmin and TN. Climix creates two separate cubes (which is correct), however, Climix only runs the index computation on one of...When running Climix with two input files from different datasets that uses two different variable names, e.g., tasmin and TN. Climix creates two separate cubes (which is correct), however, Climix only runs the index computation on one of them. This can be misleading for the user and go unnoticed. To improve the handling of this case a warning followed by aborting immediately should happen IF there are more than one cube after the concatenation AND if there is only one data variable required for the index calculation.https://git.smhi.se/climix/climix/-/issues/280new index request: number of events2023-09-12T08:41:46ZRenate Wilckenew index request: number of eventsI wish for an index that gives the "number of events", where an event would be (in my case).
Where event is defined as: (T > 95%) x number of day in a row. Where T can be tmean, tmax, or tmin.
Concrete example is (Tmax > 20C) at least 3 ...I wish for an index that gives the "number of events", where an event would be (in my case).
Where event is defined as: (T > 95%) x number of day in a row. Where T can be tmean, tmax, or tmin.
Concrete example is (Tmax > 20C) at least 3 days in a row.https://git.smhi.se/climix/climix/-/issues/281Climix API2023-06-09T14:05:16ZCarolina NilssonClimix APISome users use the functions in Climix and their own scripts which can be problematic in some cases. We need to take a decision on how one should use Climix, i.e. if we should limit and test the functions and create some sort of Climix API.Some users use the functions in Climix and their own scripts which can be problematic in some cases. We need to take a decision on how one should use Climix, i.e. if we should limit and test the functions and create some sort of Climix API.https://git.smhi.se/climix/climix/-/issues/293Allow index functions to use coordinate information2023-02-13T15:18:49ZLars BärringAllow index functions to use coordinate informationSome indices need to include coordinate information in the calculation. Examples are:
* growing season length (`gsl`) and related indices that use latitude information to shift calculation period 6 month in the southern hemisphere
* suns...Some indices need to include coordinate information in the calculation. Examples are:
* growing season length (`gsl`) and related indices that use latitude information to shift calculation period 6 month in the southern hemisphere
* sunshine duration (`SSp`) that calculates the actual sunshine duration as a fraction of the day length (latitude, time)https://git.smhi.se/climix/climix/-/issues/52Correctly implement cell_methods2023-01-25T20:16:53ZKlaus ZimmermannCorrectly implement cell_methodsThe `cell_method` information from the metadata file has to be put correctly into the output.The `cell_method` information from the metadata file has to be put correctly into the output.https://git.smhi.se/climix/climix/-/issues/295Amend index function `count_percentile_occurrences` to produce output either ...2023-04-27T06:45:46ZLars BärringAmend index function `count_percentile_occurrences` to produce output either in days or percentCLIX-META issue [#62](https://github.com/clix-meta/clix-meta/issues/62) points out that there is an inconsistency in definitions of various percentile-based temperature indices between ETCCDI and and ECA&D.
To account for this inconsis...CLIX-META issue [#62](https://github.com/clix-meta/clix-meta/issues/62) points out that there is an inconsistency in definitions of various percentile-based temperature indices between ETCCDI and and ECA&D.
To account for this inconsistency the CLIMIX index function `count_percentile_occurrences` should be able to produce results expressed either as % of calculation period (year, month, or any other), or as days according to what is provided from CLIX-META. As neither standard name nor cell method is currently available there is no complications regarding units or otherwise in that respect.
In the CLIX-META issue it is suggested that the decision on output unit could be defined by the user. Whether this should also be possible in CLIMIX should be handled in a different issue.https://git.smhi.se/climix/climix/-/issues/297Extend index function "interday_diurnal_temperature_range" to take a reducer ...2024-02-02T10:33:48ZLars BärringExtend index function "interday_diurnal_temperature_range" to take a reducer as argumentThis extension would enable e.g. `maximum` as an alternative reducer to the currently hardcoded `mean`. Such an index would be relevant as indicator of major day-to-day shifts in weather. e.g. one day overcast and only little difference ...This extension would enable e.g. `maximum` as an alternative reducer to the currently hardcoded `mean`. Such an index would be relevant as indicator of major day-to-day shifts in weather. e.g. one day overcast and only little difference between day and night temperatures, and the next day clear skies with a large difference between a warm daytime temperatures and cool nighttime temperatures.
Cf. [clix-meta #89](https://github.com/clix-meta/clix-meta/issues/89)https://git.smhi.se/climix/climix/-/issues/300Non lazy cubes throws empty assertion error2024-02-02T12:04:47ZErik HolmgrenNon lazy cubes throws empty assertion errorWe touched upon this during the last stand-up. I now came across it again working with some station data - cubes created from csv via pandas. Simply adding `cube.data = cube.lazy_data()` solves it. But maybe we should add our own more in...We touched upon this during the last stand-up. I now came across it again working with some station data - cubes created from csv via pandas. Simply adding `cube.data = cube.lazy_data()` solves it. But maybe we should add our own more informative error, or just make sure that data is lazy? Full traceback below.
```
AssertionError Traceback (most recent call last)
Cell In [122], line 1
----> 1 index([cube], client=client)
File ~/dev/climix/climix/index.py:52, in Index.__call__(self, cubes, client, sliced_mode)
50 self.index_function.prepare(cube_mapping)
51 logging.debug("Setting up aggregation")
---> 52 aggregated = multicube_aggregated_by(
53 cube_mapping,
54 coord_name,
55 self.aggregator,
56 period=self.period,
57 client=client,
58 sliced_mode=sliced_mode,
59 output_metadata=self.metadata.output,
60 )
61 aggregated.attributes["frequency"] = self.period.label
62 return aggregated
File ~/dev/climix/climix/iris.py:148, in multicube_aggregated_by(cubes, coords, aggregator, **kwargs)
145 aggregateby_cube.add_aux_coord(coord.copy(), ref_cube.coord_dims(coord))
147 # Attach the aggregate-by data into the aggregate-by cube.
--> 148 aggregateby_cube = aggregator.post_process(
149 aggregateby_cube, aggregateby_data, coords, **kwargs
150 )
152 return aggregateby_cube
File ~/dev/climix/climix/aggregators.py:65, in PointLocalAggregator.post_process(self, cube, data, coords, client, sliced_mode, **kwargs)
64 def post_process(self, cube, data, coords, client, sliced_mode, **kwargs):
---> 65 data = self.compute_pre_result(data, client, sliced_mode)
66 try:
67 post_processor = self.index_function.post_process
File ~/dev/climix/climix/aggregators.py:59, in PointLocalAggregator.compute_pre_result(self, data, client, sliced_mode)
57 logging.debug("Setting up pre-result in aggregate mode")
58 start = time.time()
---> 59 data = client.persist(data)
60 end = time.time()
61 logging.debug(f"Setup completed in {end - start:4.0f}")
File ~/miniconda3/lib/python3.10/site-packages/distributed/client.py:3437, in Client.persist(self, collections, optimize_graph, workers, allow_other_workers, resources, retries, priority, fifo_timeout, actors, **kwargs)
3434 singleton = True
3435 collections = [collections]
-> 3437 assert all(map(dask.is_dask_collection, collections))
3439 dsk = self.collections_to_dsk(collections, optimize_graph, **kwargs)
3441 names = {k for c in collections for k in flatten(c.__dask_keys__())}
AssertionError:
```https://git.smhi.se/climix/climix/-/issues/302Allow a more fine-grained division of periods than months2024-02-02T13:43:11ZLars BärringAllow a more fine-grained division of periods than monthsNow and then we have had requests for computing indices based on some other period than can be specified by whole months, e.g. counting the number of dry days between April 15 and end of June, just to mention one example.Now and then we have had requests for computing indices based on some other period than can be specified by whole months, e.g. counting the number of dry days between April 15 and end of June, just to mention one example.https://git.smhi.se/climix/climix/-/issues/303Climix configuration2024-02-02T10:33:03ZCarolina NilssonClimix configurationClimix now has a configuration file, climix_config.yml, which opens the possibility to include configurations. To allow some sort of merging for different configuration files can be beneficial in the context of a wider Climix configurati...Climix now has a configuration file, climix_config.yml, which opens the possibility to include configurations. To allow some sort of merging for different configuration files can be beneficial in the context of a wider Climix configuration handling. This needs to be further discussed.https://git.smhi.se/climix/climix/-/issues/308Check if tasmin>tasmax2023-04-20T14:33:44ZCarolina NilssonCheck if tasmin>tasmaxSome index functions relies on the fact that tasmin should be less than tasmax. This is not always the case and we need to handle this situation in some sense. E.g., index functions count_level_crossings and diurnal_temperature_range.
S...Some index functions relies on the fact that tasmin should be less than tasmax. This is not always the case and we need to handle this situation in some sense. E.g., index functions count_level_crossings and diurnal_temperature_range.
Suqgested solution: return masked or NaN values and throw some warning.https://git.smhi.se/climix/climix/-/issues/309Fix masked input data handeling in index functions2023-09-18T09:39:55ZCarolina NilssonFix masked input data handeling in index functionsThe handeling of masked input data needs to be reviewed for all of the index functions.
Here, we need to the decide on the expected behaviour of each index function. We need to decide that either, if there is one masked grid-cell the th...The handeling of masked input data needs to be reviewed for all of the index functions.
Here, we need to the decide on the expected behaviour of each index function. We need to decide that either, if there is one masked grid-cell the the output should be masked for that grid-cell or if there should be some limit for how many values that can be masked and then how this can be used in the computations. E.g., there could be some pre-processing step that checks the quality of the data and masks all values for that grid-cell which do not fulfill the conditions etc.https://git.smhi.se/climix/climix/-/issues/315Indicators calculated given a condition2023-05-08T07:51:08ZJohan SödlingIndicators calculated given a conditionIn some projects I have need of calculating indices given some condition, for example the number of zero crossings during the vegetation period, or accumulated precipitation given temperature > 0. More generally, it would be nice if Clim...In some projects I have need of calculating indices given some condition, for example the number of zero crossings during the vegetation period, or accumulated precipitation given temperature > 0. More generally, it would be nice if Climix supported calculating any index X given some condition Y, where Y is just a filter for which timesteps to use.https://git.smhi.se/climix/climix/-/issues/317Quality flag: missing data in output2024-02-02T10:29:22ZErik HolmgrenQuality flag: missing data in outputAs discussed during the technical meeting. Possibly add a flag which when toggled adds information about the amount of missing data to the output of Climix.As discussed during the technical meeting. Possibly add a flag which when toggled adds information about the amount of missing data to the output of Climix.https://git.smhi.se/climix/climix/-/issues/318Missing value threshold in config2023-05-11T13:42:06ZErik HolmgrenMissing value threshold in configAdd a config option to set the threshold for the allowed amount of missing data. This could for example allow the user to decide how many days in a running window can be missing for the calculation to still be valid.Add a config option to set the threshold for the allowed amount of missing data. This could for example allow the user to decide how many days in a running window can be missing for the calculation to still be valid.https://git.smhi.se/climix/climix/-/issues/319Consistent period specification2023-05-16T09:07:50ZCarolina NilssonConsistent period specificationThere are some differences in the period specification classes (seasonal, monthly, annual), e.g. first_month_number does only exist for annual. This can cause problems when running different index functions that utilise these features.There are some differences in the period specification classes (seasonal, monthly, annual), e.g. first_month_number does only exist for annual. This can cause problems when running different index functions that utilise these features.https://git.smhi.se/climix/climix/-/issues/324Issue when running climix API - dask issue?2023-06-15T10:02:59ZRenate WilckeIssue when running climix API - dask issue?When I run my little example script I get the following error that repeats a lot until I cancel (ctr c).
Example script:
/home/sm_renwi/Scripts/heatwavefuture/summerseason/seasonlength_paket/seasonlength/example_error_memoryview.py
/hom...When I run my little example script I get the following error that repeats a lot until I cancel (ctr c).
Example script:
/home/sm_renwi/Scripts/heatwavefuture/summerseason/seasonlength_paket/seasonlength/example_error_memoryview.py
/home/sm_renwi/Scripts/heatwavefuture/summerseason/seasonlength_paket/control_SLENS_seasonlength.yml
Error message in ipython when running "indexcube.data" after calculating indexcube:
```
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
Cell In[21], line 1
----> 1 indexcube.data
File ~/.conda/envs/climix_testconda/lib/python3.10/site-packages/iris/cube.py:2462, in Cube.data(self)
2429 @property
2430 def data(self):
2431 """
2432 The :class:`numpy.ndarray` representing the multi-dimensional data of
2433 the cube.
(...)
2460
2461 """
-> 2462 return self._data_manager.data
File ~/.conda/envs/climix_testconda/lib/python3.10/site-packages/iris/_data_manager.py:206, in DataManager.data(self)
203 if self.has_lazy_data():
204 try:
205 # Realise the lazy data.
--> 206 result = as_concrete_data(self._lazy_array)
207 # Assign the realised result.
208 self._real_array = result
File ~/.conda/envs/climix_testconda/lib/python3.10/site-packages/iris/_lazy_data.py:279, in as_concrete_data(data)
262 """
263 Return the actual content of a lazy array, as a numpy array.
264 If the input data is a NumPy `ndarray` or masked array, return it
(...)
276
277 """
278 if is_lazy_data(data):
--> 279 (data,) = _co_realise_lazy_arrays([data])
281 return data
File ~/.conda/envs/climix_testconda/lib/python3.10/site-packages/iris/_lazy_data.py:242, in _co_realise_lazy_arrays(arrays)
227 def _co_realise_lazy_arrays(arrays):
228 """
229 Compute multiple lazy arrays and return a list of real values.
230
(...)
240
241 """
--> 242 computed_arrays = da.compute(*arrays)
243 results = []
244 for lazy_in, real_out in zip(arrays, computed_arrays):
245 # Ensure we always have arrays.
246 # Note : in some cases dask (and numpy) will return a scalar
247 # numpy.int/numpy.float object rather than an ndarray.
248 # Recorded in https://github.com/dask/dask/issues/2111.
File ~/.conda/envs/climix_testconda/lib/python3.10/site-packages/dask/base.py:600, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
597 postcomputes.append(x.__dask_postcompute__())
599 results = schedule(dsk, keys, **kwargs)
--> 600 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
File ~/.conda/envs/climix_testconda/lib/python3.10/site-packages/dask/base.py:600, in <listcomp>(.0)
597 postcomputes.append(x.__dask_postcompute__())
599 results = schedule(dsk, keys, **kwargs)
--> 600 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
File ~/.conda/envs/climix_testconda/lib/python3.10/site-packages/dask/array/core.py:1283, in finalize(results)
1281 while isinstance(results2, (tuple, list)):
1282 if len(results2) > 1:
-> 1283 return concatenate3(results)
1284 else:
1285 results2 = results2[0]
File ~/.conda/envs/climix_testconda/lib/python3.10/site-packages/dask/array/core.py:5300, in concatenate3(arrays)
5298 if not ndim:
5299 return arrays
-> 5300 chunks = chunks_from_arrays(arrays)
5301 shape = tuple(map(sum, chunks))
5303 def dtype(x):
File ~/.conda/envs/climix_testconda/lib/python3.10/site-packages/dask/array/core.py:5087, in chunks_from_arrays(arrays)
5084 return (1,)
5086 while isinstance(arrays, (list, tuple)):
-> 5087 result.append(tuple(shape(deepfirst(a))[dim] for a in arrays))
5088 arrays = arrays[0]
5089 dim += 1
File ~/.conda/envs/climix_testconda/lib/python3.10/site-packages/dask/array/core.py:5087, in <genexpr>(.0)
5084 return (1,)
5086 while isinstance(arrays, (list, tuple)):
-> 5087 result.append(tuple(shape(deepfirst(a))[dim] for a in arrays))
5088 arrays = arrays[0]
5089 dim += 1
IndexError: tuple index out of range
```
Error message in terminal:
```
/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 43663 instead
warnings.warn(
/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 43577 instead
warnings.warn(
2023-06-15 10:53:40,552 - distributed.nanny - ERROR - Failed to start process
Traceback (most recent call last):
File "/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/site-packages/distributed/nanny.py", line 443, in instantiate
result = await self.process.start()
File "/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/site-packages/distributed/nanny.py", line 713, in start
await self.process.start()
File "/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/site-packages/distributed/process.py", line 55, in _call_and_set_future
res = func(*args, **kwargs)
File "/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/site-packages/distributed/process.py", line 215, in _start
process.start()
File "/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
return Popen(process_obj)
File "/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
File "/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "/home/sm_renwi/.conda/envs/climix_testconda/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
```https://git.smhi.se/climix/climix/-/issues/326The call function does not work for spell function: Spell_Length2023-06-18T12:04:47ZCarolina NilssonThe call function does not work for spell function: Spell_LengthThe call function for spell function spell_length does not work and should probably be either removed or fixed.The call function for spell function spell_length does not work and should probably be either removed or fixed.https://git.smhi.se/climix/climix/-/issues/327Investigate the behaviour of FirstOccurrence and LastOccurence2023-09-11T09:45:30ZCarolina NilssonInvestigate the behaviour of FirstOccurrence and LastOccurenceThe expected output of the index functions FirstOccurrence and LastOccurrence needs to be further investigated. FirstOccurrence call_func returns 0 for the first day. This does not match up with the output from Last Occurrence call_func ...The expected output of the index functions FirstOccurrence and LastOccurrence needs to be further investigated. FirstOccurrence call_func returns 0 for the first day. This does not match up with the output from Last Occurrence call_func that returns 1 for the first day. The output is then post process which may change the end result. Therefor, further investigation of the final output is needed to estimate if both functions are working as expected.https://git.smhi.se/climix/climix/-/issues/336Add testes for percentile functions2023-09-29T08:11:41ZCarolina NilssonAdd testes for percentile functionsThe percentile functions needs tests.The percentile functions needs tests.https://git.smhi.se/climix/climix/-/issues/338Wishlist for new indeces related to growing season2023-10-30T16:58:22ZJohan SödlingWishlist for new indeces related to growing seasonI am using Climix in some consultant projects, and for an upcoming project we would like to calculate indeces during the growing season, such as:
- The total precipitation volume (rr) during the growing season
- The number of days with m...I am using Climix in some consultant projects, and for an upcoming project we would like to calculate indeces during the growing season, such as:
- The total precipitation volume (rr) during the growing season
- The number of days with mean daily temperature > X degrees during the growing season
- The number of days with precipitation > 0 during the growing season
In a more general sense it would be nice if basically any Climix index could be calculated but only for timesteps within the vegetation periods.
I would also like it if the parameters used in calculation of the growing season were flexible (so that the user can choose the temperature and days threshold used in its calculation).