MIdAS merge requestshttps://git.smhi.se/midas/midas/-/merge_requests2024-03-15T10:24:43Zhttps://git.smhi.se/midas/midas/-/merge_requests/37Provide correct chunking information in dask_rolling_mean2024-03-15T10:24:43ZKlaus ZimmermannProvide correct chunking information in dask_rolling_meancloses #48closes #48v0.3.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/69Fixes inverse of relative humidity and component issue (Fixes #59 #83)2023-12-12T08:57:31ZWei YangFixes inverse of relative humidity and component issue (Fixes #59 #83)v0.4.0https://git.smhi.se/midas/midas/-/merge_requests/66Opt-in to datum (in sense of earth ellipsoid) support from Iris (closes #81)2022-12-12T00:59:47ZKlaus ZimmermannOpt-in to datum (in sense of earth ellipsoid) support from Iris (closes #81)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/65Remove cluster restart (closes #80)2022-12-12T00:54:49ZKlaus ZimmermannRemove cluster restart (closes #80)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/64Make zarr handling explicit to allow for storage control (closes #79)2022-12-12T00:51:30ZKlaus ZimmermannMake zarr handling explicit to allow for storage control (closes #79)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/63Add rechunking of overly fragmented calibration parameters (closes #78)2022-12-12T00:46:41ZKlaus ZimmermannAdd rechunking of overly fragmented calibration parameters (closes #78)This adds rechunking after calibration to avoid fragmentation of the calibration parameter arrays.This adds rechunking after calibration to avoid fragmentation of the calibration parameter arrays.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/62Use broadcast_to to minimize memory for empty placeholder in saving (closes #77)2022-12-12T00:40:34ZKlaus ZimmermannUse broadcast_to to minimize memory for empty placeholder in saving (closes #77)This avoids the excessive memory requirements described in #77 by exploiting numpy views. Instead of allocating and zeroing a memory block equivalent to the final result, we now only use **one timestep** of the final result, creating the...This avoids the excessive memory requirements described in #77 by exploiting numpy views. Instead of allocating and zeroing a memory block equivalent to the final result, we now only use **one timestep** of the final result, creating the larger block only as a view into this much smaller allocation.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/61Add `component` attribute to time-separated component cubes (closes #76)2022-12-12T00:36:01ZKlaus ZimmermannAdd `component` attribute to time-separated component cubes (closes #76)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/60Housekeeping (closes #75)2022-12-12T00:30:27ZKlaus ZimmermannHousekeeping (closes #75)This updates the pre-commit config and (re-)blackifies the code.This updates the pre-commit config and (re-)blackifies the code.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/59Mask handling (closes #9, #15)2022-10-24T13:59:36ZKlaus ZimmermannMask handling (closes #9, #15)This merge request adds handling of masks and unequal length timeseries. It touches topics of issues (#9, #15, #50, and #51).
I would like to solicit input on the way this is achieved. Here is what's happening:
First, for every gridpoi...This merge request adds handling of masks and unequal length timeseries. It touches topics of issues (#9, #15, #50, and #51).
I would like to solicit input on the way this is achieved. Here is what's happening:
First, for every gridpoint, Midas removes all masked points from both reference and model. If all points are masked, the output point is masked.
If after the removal the number of points is unequal between reference and model, Midas removes points from the longer array so that it shrinks to the length of the shorter one. This is done by selecting points randomly from a uniform distribution with a fixed seed for the random number generator.
While this was done to handle masks, it also deals with the situation of unequal lengths.
Do you think this is reasonable?
@a001850 @a001920 @a001497 @a001977 @a001257 ?Thomas BosshardThomas Bosshardhttps://git.smhi.se/midas/midas/-/merge_requests/58Update jobscript (closes #69)2022-09-02T10:22:14ZKlaus ZimmermannUpdate jobscript (closes #69)This removes the old non-heterogeneous jobscript and updates the heterogeneous one to deal with and make use of the recent additions to Midas and our HPC environment.This removes the old non-heterogeneous jobscript and updates the heterogeneous one to deal with and make use of the recent additions to Midas and our HPC environment.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/57Improve logging (closes #71)2022-09-02T10:12:29ZKlaus ZimmermannImprove logging (closes #71)This improves logging in several ways. It adds the creation of several log files with varying levels of detail and different components (like Midas, Dask, Distributed) to ease debugging. To support this, the command line option `--log-fi...This improves logging in several ways. It adds the creation of several log files with varying levels of detail and different components (like Midas, Dask, Distributed) to ease debugging. To support this, the command line option `--log-file` is deprecated and replaced by `--log-prefix`, which gives only the prefix to be used to distinguish the output of one Midas run from that of another. In an HPC environment, this will usually include the job number.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/56Improve output file handling (closes #49)2022-09-02T10:01:58ZKlaus ZimmermannImprove output file handling (closes #49)This changes output to initially write to a temporary file that is moved to the final destination only if and when the computations have been successful. It also adds protection from overwriting existing files that can, however, be force...This changes output to initially write to a temporary file that is moved to the final destination only if and when the computations have been successful. It also adds protection from overwriting existing files that can, however, be forced with the new `--force` (or `-f`) switch.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/55Decouple output block size and chunking in result array (closes #70)2022-09-02T09:55:00ZKlaus ZimmermannDecouple output block size and chunking in result array (closes #70)This decouples output block size and chunking of the result array, allowing for optimal choices of both.This decouples output block size and chunking of the result array, allowing for optimal choices of both.Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/54Update dask pin to >=2022.8.1 (latest) (closes #68)2022-09-01T08:53:48ZKlaus ZimmermannUpdate dask pin to >=2022.8.1 (latest) (closes #68)Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/53Add license (closes #67)2022-07-04T14:09:48ZKlaus ZimmermannAdd license (closes #67)v0.3.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/52Improve max parameter handling2022-06-22T16:21:20ZKlaus ZimmermannImprove max parameter handlingcloses #66closes #66v0.3.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/51Move loading of zarr file to adjustment step2022-06-22T16:02:28ZKlaus ZimmermannMove loading of zarr file to adjustment stepcloses #65closes #65v0.3.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/50Perform sorting on individual arrays before stacking to improve performance2022-06-22T15:57:57ZKlaus ZimmermannPerform sorting on individual arrays before stacking to improve performancecloses #64closes #64v0.3.0Klaus ZimmermannKlaus Zimmermannhttps://git.smhi.se/midas/midas/-/merge_requests/49Make smoothing parameter calculation more robust for low number of points2022-06-22T15:48:17ZKlaus ZimmermannMake smoothing parameter calculation more robust for low number of pointscloses #63closes #63v0.3.0Klaus ZimmermannKlaus Zimmermann