juanlu
10/20/2022, 10:06 AMRick Lamers
10/20/2022, 8:53 PMJerome Montino
10/21/2022, 3:52 AMRick Lamers
10/21/2022, 10:17 AMjuanlu
10/21/2022, 10:18 AMIf Polars can cleanly handle larger than memory dataframes it will be a game changer😎 https://twitter.com/RitchieVink/status/1579827660142051328
Jerome Montino
10/21/2022, 10:19 AMRick Lamers
10/21/2022, 10:20 AMTwee: more streaming queries
Rick: If Polars can cleanly handleI guess this is the essential bit. He's found a way to do it in principle, but will it be too limited (only for certain operations) or feel awkward? If he can nail it Polars is the king of the hill
Jerome Montino
10/21/2022, 10:20 AMRick Lamers
10/21/2022, 10:21 AMbetter if I can just throw my 128GB RAM at it rather than go Dask or something~Buy~rent a bigger machine >>>> more complex setup with many moving parts
Jerome Montino
10/21/2022, 10:21 AMRick Lamers
10/21/2022, 10:21 AMJerome Montino
10/21/2022, 10:22 AMRick Lamers
10/21/2022, 10:23 AMjuanlu
10/21/2022, 10:25 AMRick Lamers
10/21/2022, 10:38 AMremain a domain-specific tool for AstronomyThat seems likely.
chose HDF5 as the baselineThis feels like an achilles heel, the Arrow format brings many modern improvements to data formats. It's like a tool based on Virtual Machines vs Containers. The paradigm that will carry the next 10 years is clear.
Jerome Montino
10/22/2022, 10:06 AMRick Lamers
10/23/2022, 10:29 AMJerome Montino
11/01/2022, 9:32 AMjuanlu
11/01/2022, 9:32 AM