Re: [Python] A user friendly way to filter parquet partitions

2021-02-19 Thread Wes McKinney
We could add a section to the docs that points out some ecosystem add-on projects On Thu, Feb 18, 2021 at 10:03 PM Bill Zhao wrote: > > Hi Micah, > > Thank you for looking into this matter. > > I understand your goal of having minimal dependency and also solve > the problem from C++ for multi-lan

Re: [Python] A user friendly way to filter parquet partitions

2021-02-18 Thread Bill Zhao
Hi Micah, Thank you for looking into this matter. I understand your goal of having minimal dependency and also solve the problem from C++ for multi-language support. With that, we cannot change to use the condition package as I proposed. However, I had a difficult time making partition filtering

Re: [Python] A user friendly way to filter parquet partitions

2021-02-17 Thread Micah Kornfield
Hi Weiyang, The library looks interesting, and for python certainly seems like it might add a better user experience. I'm not super active in python maintenance (others who are can hopefully chime in). But my impression is we try to keep dependencies minimal in general. Furthermore, the goal of

[Python] A user friendly way to filter parquet partitions

2021-02-14 Thread Bill Zhao
Hi Dev team, I created a pypi package to allow user friendly expression of conditions. For example, a condition can be written as: (f.A <= 3 or f.B != 'b1') and f.C == ['c1', 'c2'] where A, B, C are partition keys, and f.C == ['c1', 'c2'] means f.C in ['c1', 'c2']. Arbitrary condition objects