GitHub user gitmodimo added a comment to the discussion: Is pyarrow.dataset still unstable?
I am not maintainer but I believe pyarrow dataset relies on C++ dataset that relies on Acero which is experimental and sadly feels abandoned. I believe Acreo should remain experimental until it handles ordered datasets and time series correctly and natively. Right now dataset ordering can only be maintained with obscure combo of single threaded scanning custom naming convention(for alphabetical ordering) and ordered sinks. It feels like Acero is just a POC to prove columnar memory format is suitable for execution and its development stopped once it was established. GitHub link: https://github.com/apache/arrow/discussions/50797#discussioncomment-17891242 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
