juanlu
Rick Lamers
Jerome Montino
10/21/2022, 3:52 AMRick Lamers
juanlu
If Polars can cleanly handle larger than memory dataframes it will be a game changerð https://twitter.com/RitchieVink/status/1579827660142051328
Jerome Montino
10/21/2022, 10:19 AMRick Lamers
Twee: more streaming queries
Rick: If Polars can cleanly handleI guess this is the essential bit. He's found a way to do it in principle, but will it be too limited (only for certain operations) or feel awkward? If he can nail it Polars is the king of the hill
Jerome Montino
10/21/2022, 10:20 AMRick Lamers
better if I can just throw my 128GB RAM at it rather than go Dask or something~Buy~rent a bigger machine >>>> more complex setup with many moving parts
Jerome Montino
10/21/2022, 10:21 AMRick Lamers
Jerome Montino
10/21/2022, 10:22 AMRick Lamers
juanlu
Rick Lamers
remain a domain-specific tool for AstronomyThat seems likely.
chose HDF5 as the baselineThis feels like an achilles heel, the Arrow format brings many modern improvements to data formats. It's like a tool based on Virtual Machines vs Containers. The paradigm that will carry the next 10 years is clear.
Jerome Montino
10/22/2022, 10:06 AMRick Lamers
Jerome Montino
11/01/2022, 9:32 AMjuanlu