Memory issue when running `python3 2_harmonize_eicu.py`

[alesssia]

Hi @USM-CHU-FGuyon ,

opening a new issue. When running python3 2_harmonize_eicu.py, I can't use sinking, and I get a memory error (see error below). I'm using a machine with 128GB of RAM.

Any suggestion on how to enable sinking or in general to solve this issue?

Many thanks again!

Collecting tsperiodic and tsaperiodic  -> Done
 --> Sinking /storage/trustalertUsersStorage/enfield/tools/BlendedICU//blended_data/long_timeseries/eicu.parquet /!\ Sinking failed.
sink_Parquet(ParquetWriteOptions { compression: Zstd(None), statistics: false, row_group_size: None, data_pagesize_limit: None, maintain_order: true }) not yet supported in standard engine. Use 'collect().write_parquet()'
       ---> Collecting...Killed

[elbg]

Hello, sorry for not seeing this issue before !
I changed institution and no longer can maintain this repository.

Indeed, the data tables are quite large, but thanks to sinking, it does run on a (reasonable) laptop.

did you use he version of polars from the env.yml (0.20) I expect it to be able to run on this version. If not, maybe you could try with a more recent polars version since sinking was improved in polars 1.*.

Best regards,
Matthieu