climix issueshttps://git.smhi.se/climix/climix/-/issues2021-07-15T15:40:59Zhttps://git.smhi.se/climix/climix/-/issues/240Improve datahandling documentation2021-07-15T15:40:59ZKlaus ZimmermannImprove datahandling documentationThe documentation overall should be improved. Borrowing from Midas, we can make a low-hanging fruit improvement in the datahandling as a first step.The documentation overall should be improved. Borrowing from Midas, we can make a low-hanging fruit improvement in the datahandling as a first step.0.14Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/239Improve iterative storage2021-07-15T15:36:11ZKlaus ZimmermannImprove iterative storageRight now, the iterative storage scheme stores the results timestep by timestep. However, due to the nature of dask arrays, every timestep requires the computation of the block that contains the timestep. Therefore, storage can be improv...Right now, the iterative storage scheme stores the results timestep by timestep. However, due to the nature of dask arrays, every timestep requires the computation of the block that contains the timestep. Therefore, storage can be improved by storing the results block-by-block instead.0.14Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/238Improve logging in datahandling2021-07-15T13:51:45ZKlaus ZimmermannImprove logging in datahandlingAt the moment, we are using mostly `logging.info` and its siblings for the logging. This works reasonably well, but could be improved by getting a specific logger instance per module as is common in Python projects.
As a first step, bec...At the moment, we are using mostly `logging.info` and its siblings for the logging. This works reasonably well, but could be improved by getting a specific logger instance per module as is common in Python projects.
As a first step, because it was already used in some debugging efforts, we can do this for the datahandling module. This should be extended to logging in all modules and combined with the efforts in !146 to make the overall logging experience more enjoyable.0.14Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/237Small technical improvements2021-07-15T13:14:04ZKlaus ZimmermannSmall technical improvementsThere are a few small technical improvements that can be added to improve the overall code-quality.
They pertain to
- Adding of potentially missing time bounds
- Removal of potentially conflicting global attributes
- Better interaction w...There are a few small technical improvements that can be added to improve the overall code-quality.
They pertain to
- Adding of potentially missing time bounds
- Removal of potentially conflicting global attributes
- Better interaction with low-level dask routines0.14Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/236Improve SLURM integration2021-07-15T13:01:24ZKlaus ZimmermannImprove SLURM integrationIt would be useful to improve the ability to run climix across several nodes on a SLURM cluster.It would be useful to improve the ability to run climix across several nodes on a SLURM cluster.0.14Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/235Update changelog for bugfix release 0.13.22021-07-14T15:58:54ZKlaus ZimmermannUpdate changelog for bugfix release 0.13.20.13.2https://git.smhi.se/climix/climix/-/issues/234KLIMATFABRIKEN outstanding issues. Deadline end July 2021.2021-10-26T13:03:14ZLars BärringKLIMATFABRIKEN outstanding issues. Deadline end July 2021.The following needs to fixed/implemented to fulfill Klimatfabriken Deliverables:
A. NOW (this week):
- [x] Bug fix: functionality to calculate indices per season (partly related to #227). For this particular project this concerns rx1day...The following needs to fixed/implemented to fulfill Klimatfabriken Deliverables:
A. NOW (this week):
- [x] Bug fix: functionality to calculate indices per season (partly related to #227). For this particular project this concerns rx1day, nzero, r10mm, r20mm. cdd.
B. Preferably this week, but can be demoted to C if need be.
- [x] Implementation of rx5day and rx{ND}day, i.e. maximum ND day precipitation. This is exactly #157, which is closely related to #86 so they should both be implemented at the same time. Clix-meta is up-to-date as far as possible (currently no CF cell-method exits, but that is not relevant.
C. Before end of month/start of your vacation:
- [x] Start/end of growing season and length of growing season.
Mention @a001284.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/233Hooks to pre-processor functions2024-02-05T13:06:31ZLars BärringHooks to pre-processor functionsExperience from using Midas in a "production" context shows that reading real-world datasets from CORDEX into midas (by analogy thus also iris) using the vanilla functions of iris causes massive failures because of shortcomings and error...Experience from using Midas in a "production" context shows that reading real-world datasets from CORDEX into midas (by analogy thus also iris) using the vanilla functions of iris causes massive failures because of shortcomings and errors in the metadata. In the wake of perfect metadata climix (and midas) should be equipped with hooks for calling a suitable set of pre-processor functions provided by the user. Tentatively, these functions would handle the following tasks:
1. [ ] Process the list of files supplied as input argument before the files are read.
2. [ ] Supplying constraints to the iris.read_raw() for subsetting the input files.
3. [ ] Process the CubeList immediately after reading and before concatenation.
4. [ ] Process attributes that differ between the individual cubes in the CubeList (and thus prevents concatenation).
5. [ ] Handle the situation that concatenation was not fully successful and did not return one cube only. This should be done in an iterative way until the result is one cube, or fail.
6. [ ] Feed processed attributrtes (from point 4) to the resulting output cube.
7. [ ] Produce a template for the output filename based on the input filenames.https://git.smhi.se/climix/climix/-/issues/232Path towards publishing climix2023-04-23T15:33:51ZLars BärringPath towards publishing climix"Publishing climix" [and clix-meta] may be divided into two partly related components:
* decision that climix [ands clix-meta] includes the functionality we want to achieve, and has the stability and maturity we want (based on whatever ..."Publishing climix" [and clix-meta] may be divided into two partly related components:
* decision that climix [ands clix-meta] includes the functionality we want to achieve, and has the stability and maturity we want (based on whatever tests we deem relevant) to tag a version 1.0 and make it publicly available (announcements how?)
* writing a scientific peer reviewed paper in a relevant journal.
These two components are somewhat abstract in relation to the hands-on work on issues and milestones in the two repositories. So, here follows an attempt at breaking down the two components to more tangible action items in relation to issues and milestones, etc. Clearly, it is my own views at this moment and should as such be regarded as a starting point for further development.
**General**
- [ ] As new issues are frequently raised, review them and make sure that they are assigned to the right milestone. Regularly review open issues.
- [ ] Get back on track having a schedule for milestones, and try to approximately stick to it. If this is not possible, review the schedule and update it.
**Ahead of releasing version 1.0**
1. [ ] Review relevant issues (and pull requests) to make sure that the two repositories are in phase with each other, and stays in phase in the future.
2. [ ] Make a list of functionality we want to have before releasing version 1. I.e follow up and update #194.
**Towards a peer review paper**
3. [ ] Very concrete schedule for releasing version 1. When submitting the manuscript version 1 should be released. I.e. this is closely related to point 1 and 2.
4. [ ] Outline what key components of the two repositories we want to focus on in the paper. I.e. from a scientific [/technical] point of what do we add?
5. [ ] Basic online documentation should be available. Could be a just better description in the repositories themselves, or more elaborate on readthedocs.io
6. [ ] We must be able to demonstrate that all indices perform produce results per the reference software (climpact 2.) for a suitable test dataset (same as delivered with climpact2).https://git.smhi.se/climix/climix/-/issues/231Improve guess_output_template2023-01-31T22:11:26ZLars BärringImprove guess_output_templateCurrently, guess_input_template goes a long way towards providing a correct output filename based on CMIP/CORDEX input files. However, handling of two filename elements need to be improved:
1. The *frequency* element is not replaced but ...Currently, guess_input_template goes a long way towards providing a correct output filename based on CMIP/CORDEX input files. However, handling of two filename elements need to be improved:
1. The *frequency* element is not replaced but instead duplicated (cf. #214).
2. If the input files are from a combination of historical and scenario simulations (as they often are), the `base` component of the filename cannot be determined resulting in a simplistic default template. There are two typical situation to handle:
* Most of the data are from the historic simulation (e.g. 1970-2005), but extended with scenario data (onwards from 2006) to at most the end of the present year. In this case it is reasonable to use the *historic files* to derive the `base` for the output template.
* Historic data is combined with scenario data that extend into the future, i.e. beyond the end of the present year. In this case it is reasonable to use the *scenario files* to derive the `base` for the output template.0.16Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/230Remove empty/inon-existing directories from input file list2021-07-06T15:10:40ZLars BärringRemove empty/inon-existing directories from input file listIf brace-expansion is used for creating the input filelist it may happen that empty/nonexisting directories are included in the list of filenames. Such directories/filenames, that have to be removed, will include the globbing character "*".If brace-expansion is used for creating the input filelist it may happen that empty/nonexisting directories are included in the list of filenames. Such directories/filenames, that have to be removed, will include the globbing character "*".Lars BärringLars Bärringhttps://git.smhi.se/climix/climix/-/issues/229Review letters used for command line arguments2023-02-16T08:24:42ZLars BärringReview letters used for command line argumentsThe letters used for one-letter command line arguments seems to have been assigned in an ad hoc fashion partly based on common practice. However, as the number of options slowly grow we might want to review, and possibly change, some of ...The letters used for one-letter command line arguments seems to have been assigned in an ad hoc fashion partly based on common practice. However, as the number of options slowly grow we might want to review, and possibly change, some of these before the package is getting used in wider circles. Currently we have:
``` bash
-h, --help
-l {debug,info,warning,error,critical}, --log-level {debug,info,warning,error,critical}
-v, --verbose
-e, --deactivate-error-tracking
-d DASK_SCHEDULER, --dask-scheduler DASK_SCHEDULER
-k, --keep-open
-s, --sliced-mode
-i, --iterative-storage
-o OUTPUT_TEMPLATE, --output OUTPUT_TEMPLATE
-x INDEX, --index INDEX
```
And there are plans (cf. #228) for an argument for replacing individual index definition elements, and for specifying index_definition file.https://git.smhi.se/climix/climix/-/issues/227Improve handling of different calculation periods (annual, seasonal, monthly)2023-01-25T16:43:29ZLars BärringImprove handling of different calculation periods (annual, seasonal, monthly)Handling of calculation period needs to be improved. In climix there are currently three open issues (#95) ~~, #170, and #226), and one over at clix-meta [#29](https://github.com/clix-meta/clix-meta/issues/29)~~ that point at various sho...Handling of calculation period needs to be improved. In climix there are currently three open issues (#95) ~~, #170, and #226), and one over at clix-meta [#29](https://github.com/clix-meta/clix-meta/issues/29)~~ that point at various shortcomings, vagueness, and bugs in the current functionality.
Issue #255 might also relate here.0.16Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/226wetdays djf error on the maxium number of days2021-07-22T14:40:57ZRamon Fuentes Francowetdays djf error on the maxium number of daysWhen using a modified version of the wetdays for djf, the maximum number of days that comes out in the output file is 30, I suspect it is only summing in one month, and not on the entire djf season.
See below the modified version I used...When using a modified version of the wetdays for djf, the maximum number of days that comes out in the output file is 30, I suspect it is only summing in one month, and not on the entire djf season.
See below the modified version I used:
```yaml
wetdays_djf:
reference: CLIPC
period:
allowed:
annual:
seasonal: 'djf'
monthly:
default: seasonal
output:
var_name: "wetdays"
standard_name: number_of_days_with_lwe_thickness_of_precipitation_amount_above_threshold
proposed_standard_name: number_of_occurrences_with_lwe_thickness_of_precipitation_amount_at_or_above_threshold
long_name: "Number of Wet Days (precip >= 1 mm)"
units: "days"
cell_methods:
- time: sum within days
- time: sum over days
input:
data: pr
index_function:
name: count_occurrences
parameters:
threshold:
kind: quantity
standard_name: lwe_precipitation_rate
long_name: "Wet day threshold"
data: 1
units: "mm day-1"
condition:
kind: operator
operator: ">"
ET:
short_name:
long_name:
definition:
comment:
```0.14https://git.smhi.se/climix/climix/-/issues/225Replace pkg_resources.iter_entry_points with importlib.metadata.entry_points2021-07-16T11:23:28ZKlaus ZimmermannReplace pkg_resources.iter_entry_points with importlib.metadata.entry_pointsThe older `pkg_resources` package has some performance deficiencies when it comes to handling entry_points. The newer `importlib.metadata` package, built on top of the new importlib infrastructure in Python 3, performs better and is reco...The older `pkg_resources` package has some performance deficiencies when it comes to handling entry_points. The newer `importlib.metadata` package, built on top of the new importlib infrastructure in Python 3, performs better and is recommended for future use. While it has been integrated into the standard library with Python 3.8, a sufficiently recent version is only integrated from Python 3.10 onwards. In the meantime, we rely on the third-party package `importlib_metadata` with an import guard to make the functionality available.0.14https://git.smhi.se/climix/climix/-/issues/224CountPercentileOccurrences doesn't respect the condition operator2021-05-04T11:56:15ZKlaus ZimmermannCountPercentileOccurrences doesn't respect the condition operatorOne of the issues that surfaced in #205 is that `CountPercentileOccurrences` doesn't respect the condition operator, i.e. it always checks that the data is less than the threshold, regardless of the operator given in the index definition...One of the issues that surfaced in #205 is that `CountPercentileOccurrences` doesn't respect the condition operator, i.e. it always checks that the data is less than the threshold, regardless of the operator given in the index definition. This should be corrected.0.13.2Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/223Add pre-commit hooks2021-07-16T10:30:02ZKlaus ZimmermannAdd pre-commit hooksProviding a [pre-commit](https://pre-commit.com/) configuration will allow us consistent code quality already at the time of commit. It can be used directly by developers using pre-commit hooks, but might also be integrated into the CI p...Providing a [pre-commit](https://pre-commit.com/) configuration will allow us consistent code quality already at the time of commit. It can be used directly by developers using pre-commit hooks, but might also be integrated into the CI pipelines using the same configuration later on.0.14https://git.smhi.se/climix/climix/-/issues/220Pin iris also in environment.yml2021-04-14T15:10:01ZKlaus ZimmermannPin iris also in environment.ymlThe pinning of iris (see #218) should happen also in the `environment.yml` file.The pinning of iris (see #218) should happen also in the `environment.yml` file.0.13.2Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/219Update to iris 32021-07-14T16:08:08ZKlaus ZimmermannUpdate to iris 3We should update to the new iris version 3. This might require some changes to climix.We should update to the new iris version 3. This might require some changes to climix.https://git.smhi.se/climix/climix/-/issues/218Pin iris version2021-04-14T15:07:43ZKlaus ZimmermannPin iris versionThe current code is not compatible with iris 3, so we should pin to iris<3 to guarantee a working code.The current code is not compatible with iris 3, so we should pin to iris<3 to guarantee a working code.0.13.2Klaus ZimmermannKlaus Zimmermann