climix issueshttps://git.smhi.se/climix/climix/-/issues2024-03-13T10:36:03Zhttps://git.smhi.se/climix/climix/-/issues/367Investigate iterative storage bevahoiur when running jobscripts2024-03-13T10:36:03ZJoakim LöwInvestigate iterative storage bevahoiur when running jobscriptsWhen running either of the jobscripts on freja, enabling the iterative storage flag (`-i`) results in a file with considerably smaller file size, and the log suggests only first slice is saved. Investigate the reason for this.When running either of the jobscripts on freja, enabling the iterative storage flag (`-i`) results in a file with considerably smaller file size, and the log suggests only first slice is saved. Investigate the reason for this.https://git.smhi.se/climix/climix/-/issues/346Fix Numba error in spell function tests2024-02-21T10:30:20ZJoakim LöwFix Numba error in spell function testsRunning the tests (`test_spell_functions`) throws an error from Numba. Has to do with undefined variables.Running the tests (`test_spell_functions`) throws an error from Numba. Has to do with undefined variables.https://git.smhi.se/climix/climix/-/issues/341add pandas to requirements2023-11-15T17:00:57ZLars Bärringadd pandas to requirementsI tried to update from a really old install and then got the error:
```bash
>pip install -e .
Obtaining file:///home/sm_lbarr/CODE/climix
Installing build dependencies ... done
Checking if build backend supports build_editable ... d...I tried to update from a really old install and then got the error:
```bash
>pip install -e .
Obtaining file:///home/sm_lbarr/CODE/climix
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Preparing editable metadata (pyproject.toml) ... done
...
Building wheels for collected packages: climix
Building editable for climix (pyproject.toml) ... done
...
Successfully installed bottleneck-1.3.7 climix-0.20.0.dev3+g6efeda6 heapdict-1.0.1 isodate-0.6.1 msgpack-1.0.7 psutil-5.9.6 sortedcontainers-2.4.0 tblib-3.0.0 tornado-6.3.3
>climix -h
Traceback (most recent call last):
File "/home/sm_lbarr/.conda/envs/climix-devel/bin/climix", line 5, in <module>
from climix.main import main
File "/home/sm_lbarr/CODE/climix/climix/main.py", line 16, in <module>
from .datahandling import prepare_input_data, save
File "/home/sm_lbarr/CODE/climix/climix/datahandling.py", line 16, in <module>
from .util import find_cube_differences
File "/home/sm_lbarr/CODE/climix/climix/util/__init__.py", line 4, in <module>
from .cube_diffs import find_cube_differences
File "/home/sm_lbarr/CODE/climix/climix/util/cube_diffs.py", line 7, in <module>
import pandas as pd
ModuleNotFoundError: No module named 'pandas'
```
A manual install of pandas did the trick.0.20 (Urbane Goat)Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/325Error when integrations tests are generated2023-11-05T15:29:56ZJoakim LöwError when integrations tests are generatedIntegrations tests currently cannot be run. Pytest returns the following error when running from command line:
```
====================================================================== test session starts ==============================...Integrations tests currently cannot be run. Pytest returns the following error when running from command line:
```
====================================================================== test session starts ======================================================================
platform linux -- Python 3.10.11, pytest-7.3.2, pluggy-1.0.0
rootdir: /home/sm_joalo/dev/repos/climix
configfile: pyproject.toml
testpaths: tests
collected 213 items / 1 error
============================================================================ ERRORS =============================================================================
______________________________________________________ ERROR collecting tests/integration/test_indices.py _______________________________________________________
tests/integration/test_indices.py:64: in <module>
generate_test_index_parametrization(),
tests/integration/test_indices.py:45: in generate_test_index_parametrization
config = read_test_configuration()
tests/integration/conftest.py:14: in read_test_configuration
config_string = files("tests.integration").joinpath("configuration.yml").read_text()
../../../.conda/envs/climix/lib/python3.10/importlib/_common.py:22: in files
return from_package(get_package(package))
../../../.conda/envs/climix/lib/python3.10/importlib/_common.py:67: in get_package
if wrap_spec(resolved).submodule_search_locations is None:
../../../.conda/envs/climix/lib/python3.10/importlib/_adapters.py:16: in __getattr__
return getattr(self.spec, name)
E AttributeError: 'NoneType' object has no attribute 'submodule_search_locations'
==================================================================== short test summary info ====================================================================
ERROR tests/integration/test_indices.py - AttributeError: 'NoneType' object has no attribute 'submodule_search_locations'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================= 1 error in 24.99s =======================================================================
```0.20 (Urbane Goat)Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/335Use numpys array_equal function in tests2023-09-21T10:11:04ZCarolina NilssonUse numpys array_equal function in testsWhen having tests that produces nan-values as a valid result we want to be able to test this. As it is right now comparing two nan-values will result in not equal. To be able of comparing nan-values we can switch to using numpy function ...When having tests that produces nan-values as a valid result we want to be able to test this. As it is right now comparing two nan-values will result in not equal. To be able of comparing nan-values we can switch to using numpy function array_equal and set input parameter equal_nan = True. This function also compares shapes of the result and the data values underneath masked values. Switching to the array_equal function generates some errors in the tests listed bellow:
- [x] FirstOccurrence and LastOccurrence, call/lazy function last test generates an error due to inaccurate shape of expected output. (Error in test parameters)
- [x] Percentile and ThresholdedPercentile, call function all tests returns an inaccurate shape of result. (Error in call function)
- [x] ThresholdedRunningStatistics, call/lazy function tests 2 and 4 returns an error due to wrong data value underneath the mask in the expected output. (Error in test parameters)0.20 (Urbane Goat)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/322Numba 0.57.0 causes error for masked array2023-09-13T13:00:55ZJoakim LöwNumba 0.57.0 causes error for masked arrayWhen running climix, numba throws an error for some cases (see #321). I suggest setting the numba version to `- numba<0.57` in the `environment.yml` for the next release:
```
2023-06-12 10:42:39,248 - distributed.worker - WARNING - Compu...When running climix, numba throws an error for some cases (see #321). I suggest setting the numba version to `- numba<0.57` in the `environment.yml` for the next release:
```
2023-06-12 10:42:39,248 - distributed.worker - WARNING - Compute Failed
Key: ('chunk-2ad0dae06c51d593e75de7dbfe6cb672', 0, 0, 0)
Function: subgraph_callable-d585f943-ba41-4885-8faa-eb2cba41
args: (masked_array(
data=[[[--, --, --, ..., 278.2850036621094, 277.3859558105469,
276.8891296386719],
[--, 279.6990661621094, 279.4710998535156, ...,
278.3681335449219, 277.4391784667969, 277.0628967285156],
[--, --, --, ..., 277.3985900878906, 276.5505065917969,
276.6014709472656],
...,
[--, --, --, ..., 265.25921630859375, 265.71435546875,
265.8021240234375],
[--, --, --, ..., 264.7350158691406, 265.0195007324219,
265.303955078125],
[--, --, --, ..., 262.71636962890625, 263.4328308105469,
264.1492614746094]],
[[--, --, --, ..., 276.0057067871094, 275.7838439941406,
275.8836364746094],
[--, 277.0145568847656, 275.7981262207031, ...,
274.5687561035156, 274.2306213378906, 274.3453674316406],
[--, --, --, ..., 273.7345275878906, 273.2975158691406,
273.5181579589844],
...,
[--, --, --, ..., 262.7914733886
kwargs: {}
Exception: "NumbaTypeError('\\x1b[1mUnsupported array type: numpy.ma.MaskedArray.\\x1b[0m')"
```0.19 (Poco Mas)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/316"Requested dask.distributed scheduler but no Client active." RuntimeError for...2023-09-08T12:49:09ZCarolina Nilsson"Requested dask.distributed scheduler but no Client active." RuntimeError for larger computations1. Installing a new environment: `mamba create -n myenv climix`
2. activating the env and running: `climix -e -x tn10p /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_...1. Installing a new environment: `mamba create -n myenv climix`
2. activating the env and running: `climix -e -x tn10p /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20060101-20101231.nc /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20110101-20151231.nc -r 2007/2009`
Returns the following RuntimeError and saves no result:
```
101637ms:main.py:main() INFO:root:Calculation took 94.1128 seconds.
2023-05-08 12:44:25,748 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:33317 -> tcp://127.0.0.1:34451
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/worker.py", line 1787, in get_data
response = await comm.read(deserializers=serializers)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://127.0.0.1:33317 remote=tcp://127.0.0.1:45308>: Stream is closed
2023-05-08 12:44:25,749 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:33317 -> tcp://127.0.0.1:46206
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/worker.py", line 1787, in get_data
response = await comm.read(deserializers=serializers)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://127.0.0.1:33317 remote=tcp://127.0.0.1:46116>: Stream is closed
2023-05-08 12:44:25,793 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:36001 -> tcp://127.0.0.1:34451
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/worker.py", line 1787, in get_data
response = await comm.read(deserializers=serializers)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://127.0.0.1:36001 remote=tcp://127.0.0.1:33512>: Stream is closed
2023-05-08 12:44:25,795 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:36001 -> tcp://127.0.0.1:46206
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/worker.py", line 1787, in get_data
response = await comm.read(deserializers=serializers)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://127.0.0.1:36001 remote=tcp://127.0.0.1:34318>: Stream is closed
2023-05-08 12:44:25,800 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:44554 -> tcp://127.0.0.1:34451
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/worker.py", line 1787, in get_data
response = await comm.read(deserializers=serializers)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://127.0.0.1:44554 remote=tcp://127.0.0.1:39332>: Stream is closed
2023-05-08 12:44:25,801 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:44554 -> tcp://127.0.0.1:46206
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/worker.py", line 1787, in get_data
response = await comm.read(deserializers=serializers)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://127.0.0.1:44554 remote=tcp://127.0.0.1:40084>: Stream is closed
2023-05-08 12:44:25,832 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:36220 -> tcp://127.0.0.1:34451
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/worker.py", line 1787, in get_data
response = await comm.read(deserializers=serializers)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://127.0.0.1:36220 remote=tcp://127.0.0.1:43522>: Stream is closed
2023-05-08 12:44:25,833 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:36220 -> tcp://127.0.0.1:46206
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/worker.py", line 1787, in get_data
response = await comm.read(deserializers=serializers)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://127.0.0.1:36220 remote=tcp://127.0.0.1:43490>: Stream is closed
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-conda/bin/climix", line 10, in <module>
sys.exit(main())
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/climix/main.py", line 353, in main
do_main(
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/climix/main.py", line 325, in do_main
save(
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/climix/datahandling.py", line 371, in save
result.data = r.result()
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/distributed/client.py", line 317, in result
raise exc.with_traceback(tb)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/dask/optimization.py", line 990, in __call__
return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/dask/core.py", line 149, in get
result = _execute_task(task, cache)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/dask/core.py", line 119, in _execute_task
return func(*(_execute_task(a, cache) for a in args))
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/dask/utils.py", line 73, in apply
return func(*args, **kwargs)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/dask/array/chunk.py", line 225, in argtopk
if abs(k) >= a.shape[axis]:
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/dask/array/core.py", line 1868, in __bool__
return bool(self.compute())
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/dask/base.py", line 314, in compute
(result,) = compute(self, traverse=False, **kwargs)
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/dask/base.py", line 587, in compute
schedule = get_scheduler(
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/dask/base.py", line 1400, in get_scheduler
return get_scheduler(scheduler=config.get("scheduler", None))
File "/home/sm_carni/.conda/envs/climix-conda/lib/python3.10/site-packages/dask/base.py", line 1375, in get_scheduler
raise RuntimeError(
RuntimeError: Requested dask.distributed scheduler but no Client active.
```
3. Running another smaller index: `climix -e -x tn /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20060101-20101231.nc /nobackup/rossby27/users/sm_carni/data/tmp/data_files/tasmin_EUR-11_MPI-M-MPI-ESM-LR_rcp85_r2i1p1_MPI-CSC-REMO2009_v1_day_20110101-20151231.nc`
-----> Returns no error.
4. Downgrading dask to `mamba install dask==2023.4.0`, solves this error. But, results in another error when running a simpler index. Running `climix -e -x txx /home/rossby/data_lib/esgf/cordex/output/EUR-11/SMHI/NCC-NorESM1-M/rcp85/r1i1p1/RCA4/v1/day/tasmax/latest/tasmax_EUR-11_NCC-NorESM1-M_rcp85_r1i1p1_SMHI-RCA4_v1_day_20060101-20101231.nc /home/rossby/data_lib/esgf/cordex/output/EUR-11/SMHI/NCC-NorESM1-M/rcp85/r1i1p1/RCA4/v1/day/tasmax/latest/tasmax_EUR-11_NCC-NorESM1-M_rcp85_r1i1p1_SMHI-RCA4_v1_day_20110101-20151231.nc` gives:
```
INFO:distributed.scheduler:Lost all workers
INFO:distributed.batched:Batched Comm Closed <TCP (closed) Scheduler connection to worker local=tcp://127.0.0.1:36766 remote=tcp://127.0.0.1:40786>
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-latest/lib/python3.10/site-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File "/home/sm_carni/.conda/envs/climix-latest/lib/python3.10/site-packages/tornado/gen.py", line 767, in run
value = future.result()
File "/home/sm_carni/.conda/envs/climix-latest/lib/python3.10/site-packages/distributed/comm/tcp.py", line 269, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
INFO:distributed.batched:Batched Comm Closed <TCP (closed) Scheduler connection to worker local=tcp://127.0.0.1:36766 remote=tcp://127.0.0.1:40776>
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix-latest/lib/python3.10/site-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File "/home/sm_carni/.conda/envs/climix-latest/lib/python3.10/site-packages/tornado/gen.py", line 767, in run
value = future.result()
File "/home/sm_carni/.conda/envs/climix-latest/lib/python3.10/site-packages/distributed/comm/tcp.py", line 269, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
```0.19 (Poco Mas)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/307RunningStatistics and ThresholdedRunningStatistics does not pass all unit tests2023-07-03T15:50:06ZCarolina NilssonRunningStatistics and ThresholdedRunningStatistics does not pass all unit testsThe RunningStatistics and ThresholdedRunningStatistics index functions does not pass all the unit tests. Some test fails because the mask is not preserved in the process e.g., when using np.concatenate and np.where. Other tests fails if ...The RunningStatistics and ThresholdedRunningStatistics index functions does not pass all the unit tests. Some test fails because the mask is not preserved in the process e.g., when using np.concatenate and np.where. Other tests fails if another statistics than "max" is used and some fails because of the padding with zeros in the start and end which can give lower aggregated values.0.19 (Poco Mas)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/310Percentile and ThresholdedPercentile index functions does not pass all unit t...2023-06-19T14:40:11ZCarolina NilssonPercentile and ThresholdedPercentile index functions does not pass all unit testsThe Percentile and ThresholdedPercentile index functions does not pass all unit tests.
1. The percentile call_function does not run since the aux coordinate for the percentile is passed as an input to the numpy function and not the poin...The Percentile and ThresholdedPercentile index functions does not pass all unit tests.
1. The percentile call_function does not run since the aux coordinate for the percentile is passed as an input to the numpy function and not the point value of the aux coordinate.
2. The numpy percentile function does not take the mask into consideration when computing the percentiles. A work around is to set the thresholded values to NaN and then use the np.nanpercentile to exclude the masked values from the computation.0.19 (Poco Mas)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/305CountJointOccurrences index functions does not pass all unit tests2023-05-24T14:39:03ZCarolina NilssonCountJointOccurrences index functions does not pass all unit testsWhen running count_joint_occurrences_precipitation_temperature, where one of the inputs have masked data, will return the other condition as True or False.
Here the mask needs to be preserved, if a grid-cell contains masked data then th...When running count_joint_occurrences_precipitation_temperature, where one of the inputs have masked data, will return the other condition as True or False.
Here the mask needs to be preserved, if a grid-cell contains masked data then the output grid-cell should probably be masked as well.0.19 (Poco Mas)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/313cube_diffs table output is not working with pandas 2.02023-05-15T12:58:00ZJoakim Löwcube_diffs table output is not working with pandas 2.0With `pandas 2.0` the following error is raised when two datasets can not be combined:
```
Traceback (most recent call last):
File "/home/sm_joalo/.conda/envs/climix-test/bin/climix", line 8, in <module>
sys.exit(main())
File "/h...With `pandas 2.0` the following error is raised when two datasets can not be combined:
```
Traceback (most recent call last):
File "/home/sm_joalo/.conda/envs/climix-test/bin/climix", line 8, in <module>
sys.exit(main())
File "/home/sm_joalo/dev/repos/climix/climix/main.py", line 353, in main
do_main(
File "/home/sm_joalo/dev/repos/climix/climix/main.py", line 316, in do_main
input_data = prepare_input_data(datafiles, climix_config)
File "/home/sm_joalo/dev/repos/climix/climix/datahandling.py", line 266, in prepare_input_data
find_cube_differences(
File "/home/sm_joalo/dev/repos/climix/climix/util/cube_diffs.py", line 330, in find_cube_differences
print_dataframe(dataframe, var_name)
File "/home/sm_joalo/dev/repos/climix/climix/util/cube_diffs.py", line 296, in print_dataframe
with pd.option_context('display.max_colwidth', MAX_COL_WIDTH,
File "/home/sm_joalo/.conda/envs/climix-test/lib/python3.10/site-packages/pandas/_config/config.py", line 441, in __enter__
self.undo = [(pat, _get_option(pat, silent=True)) for pat, val in self.ops]
File "/home/sm_joalo/.conda/envs/climix-test/lib/python3.10/site-packages/pandas/_config/config.py", line 441, in <listcomp>
self.undo = [(pat, _get_option(pat, silent=True)) for pat, val in self.ops]
File "/home/sm_joalo/.conda/envs/climix-test/lib/python3.10/site-packages/pandas/_config/config.py", line 135, in _get_option
key = _get_single_key(pat, silent)
File "/home/sm_joalo/.conda/envs/climix-test/lib/python3.10/site-packages/pandas/_config/config.py", line 121, in _get_single_key
raise OptionError(f"No such keys(s): {repr(pat)}")
pandas._config.config.OptionError: No such keys(s): 'display.column_space'
```
Expected output would be the table of cube differences.
It seems the `display.column_space` option, used in `print_dataframe(...)`, has been deprecated and removed in `pandas 2.0` (I believe we have used `pandas 1.5.x` before).0.19 (Poco Mas)Carolina NilssonCarolina Nilssonhttps://git.smhi.se/climix/climix/-/issues/312Binary dist is missing etc directory and data files2023-04-25T13:22:24ZKlaus ZimmermannBinary dist is missing etc directory and data filesIn the move to pyproject.toml, we lost the crucial data files from the binary distribution.
While this does not affect development installs, it does affect wheels and other copying installations.In the move to pyproject.toml, we lost the crucial data files from the binary distribution.
While this does not affect development installs, it does affect wheels and other copying installations.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/301Running climix 0.17 with new env throws hdf5 error messages2023-04-19T13:24:26ZCarolina NilssonRunning climix 0.17 with new env throws hdf5 error messagesCreating a new env using climix 0.17 environment.yml and running climix throws hdf5 error messages.
```
HDF5-DIAG: Error detected in HDF5 (1.14.0) thread 1:
#000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attrib...Creating a new env using climix 0.17 environment.yml and running climix throws hdf5 error messages.
```
HDF5-DIAG: Error detected in HDF5 (1.14.0) thread 1:
#000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute
major: Attribute
minor: Can't open object
#001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeBitRoundNumberOfSignificantBits'
major: Attribute
minor: Can't open object
#002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeBitRoundNumberOfSignificantBits'
major: Attribute
minor: Can't open object
#003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#005: H5VLnative_attr.c line 161 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#006: H5Aint.c line 658 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#007: H5Oattribute.c line 502 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'
major: Attribute
minor: Object not found
.
.
.
```
This can be solved by adding a constraint in the environment.yml `libnetcdf<4.9.1` Which makes the following changes:
```
──────────────────────────────────────────────────────────────────────────────────────
Install:
──────────────────────────────────────────────────────────────────────────────────────
+ jpeg 9e h166bdaf_2 conda-forge/linux-64 Cached
Change:
──────────────────────────────────────────────────────────────────────────────────────
- hdf4 4.2.15 h501b40f_6 conda-forge
+ hdf4 4.2.15 h9772cbc_5 conda-forge/linux-64 Cached
- lcms2 2.15 haa2dc70_1 conda-forge
+ lcms2 2.15 hfd0df8a_0 conda-forge/linux-64 Cached
- libtiff 4.5.0 ha587672_6 conda-forge
+ libtiff 4.5.0 h6adf6a1_2 conda-forge/linux-64 Cached
- pillow 9.4.0 py310h065c6d2_2 conda-forge
+ pillow 9.4.0 py310h023d228_1 conda-forge/linux-64 Cached
Downgrade:
──────────────────────────────────────────────────────────────────────────────────────
- hdf5 1.14.0 nompi_hb72d44e_103 conda-forge
+ hdf5 1.12.2 nompi_h4df4325_101 conda-forge/linux-64 Cached
- libdeflate 1.18 h0b41bf4_0 conda-forge
+ libdeflate 1.17 h0b41bf4_0 conda-forge/linux-64 Cached
- libjpeg-turbo 2.1.5.1 h0b41bf4_0 conda-forge
+ libjpeg-turbo 2.1.4 h166bdaf_0 conda-forge/linux-64 Cached
- libnetcdf 4.9.2 nompi_hf3f8848_103 conda-forge
+ libnetcdf 4.8.1 nompi_h261ec11_106 conda-forge/linux-64 Cached
- netcdf4 1.6.3 nompi_py310h2d0b64f_102 conda-forge
+ netcdf4 1.6.2 nompi_py310h55e1e36_100 conda-forge/linux-64 Cached
Summary:
Install: 1 packages
Change: 4 packages
Downgrade: 5 packages
```
However constraining libnetcdf, seems to create a new error with dask:
```
INFO:distributed.batched:Batched Comm Closed <TCP (closed) Scheduler connection to worker local=tcp://127.0.0.1:49048 remote=tcp://127.0.0.1:53328>
Traceback (most recent call last):
File "/home/sm_carni/.conda/envs/climix_error_test/lib/python3.10/site-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File "/home/sm_carni/.conda/envs/climix_error_test/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
value = future.result()
File "/home/sm_carni/.conda/envs/climix_error_test/lib/python3.10/site-packages/distributed/comm/tcp.py", line 269, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
```
can be solved by constraining `netCDF4==1.6.0` but I am not sure if this is the right approach.0.18 (Gull Olle)Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/267Difference in output when storing iterative using flag -i and storing non-ite...2023-03-13T10:19:37ZCarolina NilssonDifference in output when storing iterative using flag -i and storing non-iterative on masked input dataThere is an difference in the output using iterative-storage and non-iterative storage, i.e., with or without the flag -i, on masked input data.
E.g.,
`climix -e -v -i -o /path/to/output/file.nc /home/rossby/joint_exp/climix/1/testdata...There is an difference in the output using iterative-storage and non-iterative storage, i.e., with or without the flag -i, on masked input data.
E.g.,
`climix -e -v -i -o /path/to/output/file.nc /home/rossby/joint_exp/climix/1/testdata/NGCD/annual/Aland/NGCD_RR*.nc -x r10mm`
In the output the masked area is filled with value 0. Whereas, for
`climix -e -v -o /path/to/output/file.nc /home/rossby/joint_exp/climix/1/testdata/NGCD/annual/Aland/NGCD_RR*.nc -x r10mm`
the masked area is still masked, i.e, filled with 1e+20.
Loading the output cubes and looking at the data, there seems to be a difference where one has a defined mask with shape (49, 75, 56) and the other one has the default mask=False with shape ().0.17 (Black Lion)Erik HolmgrenErik Holmgrenhttps://git.smhi.se/climix/climix/-/issues/256Consecutive index fails on 1 dimensional cubes2023-03-13T09:32:26ZErik HolmgrenConsecutive index fails on 1 dimensional cubesThe index `ctxge{TT}` fails on 1-dimensional cubes (i.e. cubes with only a time dimension, spatial averages etc.). Klaus did some digging and we found that the `spell_kernel` expects a 3-dimensional cube. A quick fix is to add dummy coor...The index `ctxge{TT}` fails on 1-dimensional cubes (i.e. cubes with only a time dimension, spatial averages etc.). Klaus did some digging and we found that the `spell_kernel` expects a 3-dimensional cube. A quick fix is to add dummy coordinates to such a cube before computing the index, but it would still be good if these types of indices worked on all cubes.
I haven't tried all the consecutive indices but I assume this issue is true for all of them. To get the true "exception" trying this, one has to disable numba in the `spell_kernel`.0.17 (Black Lion)Erik HolmgrenErik Holmgrenhttps://git.smhi.se/climix/climix/-/issues/260Rx2day, Rx5day resulting in some infinite values2023-01-31T14:59:28ZErik HolmgrenRx2day, Rx5day resulting in some infinite valuesTwo extreme precipitation indices, `rx2day` and `rx5day`, produces some infinite values on e.g. GridClim and PTHBV datasets. For both these datasets there are no issues with `rx1day`.
Example (on bi):
```python
import iris
from climix....Two extreme precipitation indices, `rx2day` and `rx5day`, produces some infinite values on e.g. GridClim and PTHBV datasets. For both these datasets there are no issues with `rx1day`.
Example (on bi):
```python
import iris
from climix.metadata import load_metadata
from dask.distributed import Client
client = Client()
fname = "/nobackup/rossby26/users/sm_erhol/extremeEventAttribution/gavle2021/pr_gavle2021_SMHIGridClim_day_19610101-20181230.nc"
# This will be an masked array
cube = iris.load_cube(fname)
index_catalogue = load_metadata()
index = index_catalogue.prepare_indices(["rx2day"])[0]
index_cube = index([cube], client)
assert not np.any(np.isinf(index_cube.data))
```
When realising the `index_cube` some overflow warnings are thrown. If the spatial dimensions of the cube are collapsed (average, max), no infinite values are produced.0.16Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/climix/climix/-/issues/265ThresholdedRunningStatistics lazy_func calls call_func of RunningStatistics2022-12-07T14:57:22ZJoakim LöwThresholdedRunningStatistics lazy_func calls call_func of RunningStatistics`lazy_func` implementation in `ThresholdedRunningStatistics` calls `call_func` of inherited `RunningStatistics`. Should probably call `RunningStatistics.lazy_func` instead. Change and make manual test to confirm.`lazy_func` implementation in `ThresholdedRunningStatistics` calls `call_func` of inherited `RunningStatistics`. Should probably call `RunningStatistics.lazy_func` instead. Change and make manual test to confirm.0.16Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/211Fix output units for index function count_occurrences2022-11-16T08:00:57ZLars BärringFix output units for index function count_occurrencesIt seems that the index function `count_occurrences` expect output to have units `day`. This is however the `proposed_unit`, which not consistent with current the current canonical unit `1` as defined by the `standard names`. This has pr...It seems that the index function `count_occurrences` expect output to have units `day`. This is however the `proposed_unit`, which not consistent with current the current canonical unit `1` as defined by the `standard names`. This has probably remained under the radar because I was until recently using a really old version of `index_definition.yml`. If simple to fix then target bugfix milestone 13.2, else postpone it to a later milestone.0.15Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/215Climix does not gracefully handle single node / singe core2022-09-15T12:18:50ZLars BärringClimix does not gracefully handle single node / singe coreClimix crashes when running on a shared node with `interactive -N 1 -n 1`:
```
>climix -s -e -x txx /home/rossby/prod/201137/netcdf/day/tasmax_EUR-11_ICHEC-EC-EARTH_historical_r12i1p1_SMHI-RCA4_v1_day_*.nc
INFO:root:Loading metadata
T...Climix crashes when running on a shared node with `interactive -N 1 -n 1`:
```
>climix -s -e -x txx /home/rossby/prod/201137/netcdf/day/tasmax_EUR-11_ICHEC-EC-EARTH_historical_r12i1p1_SMHI-RCA4_v1_day_*.nc
INFO:root:Loading metadata
Traceback (most recent call last):
File "/home/sm_lbarr/.conda/envs/climix-devel-3/bin/climix", line 11, in <module>
load_entry_point('climix', 'console_scripts', 'climix')()
File "/home/sm_lbarr/CODE/climix/climix/main.py", line 146, in main
with setup_scheduler(args) as scheduler:
File "/home/sm_lbarr/CODE/climix/climix/dask_setup.py", line 160, in setup_scheduler
return scheduler(**scheduler_kwargs)
File "/home/sm_lbarr/CODE/climix/climix/dask_setup.py", line 68, in __init__
memory_limit = (system.MEMORY_LIMIT*.9) / n_workers
ZeroDivisionError: float division by zero
```
Preferably it should instead give a graceful error message.0.16Joakim LöwJoakim Löwhttps://git.smhi.se/climix/climix/-/issues/253Newer version of Iris lazy-load coord data, causes crash in climix during save2022-09-14T09:42:42ZJoakim LöwNewer version of Iris lazy-load coord data, causes crash in climix during saveIn a newer version of Iris coord data of a cube is lazy-loaded. Therefore, when saving output in climix, the data may not exist in memory yet, which causes a crash. Solution: touch the coord data before saving.In a newer version of Iris coord data of a cube is lazy-loaded. Therefore, when saving output in climix, the data may not exist in memory yet, which causes a crash. Solution: touch the coord data before saving.Joakim LöwJoakim Löw