Re: [DISCUSS] Arrow array representation of statistics

2024-12-22 Thread Sutou Kouhei
Hi, There are no objections how to process TODOs. I've used suggested approaches for them. I've added examples for array. I'll start a vote for this again. Thanks, -- kou In <20241218.170107.1224859230223769469@clear-code.com> "Re: [DISCUSS] Arrow array repres

Re: [DISCUSS] Arrow array representation of statistics

2024-12-18 Thread Sutou Kouhei
Hi, Thanks for sharing your opinion. There is no objection to remove the C data interface limitation. I opened a new PR that focus on only statistics schema: * PR: https://github.com/apache/arrow/pull/45058 * Preview: http://crossbow.voltrondata.com/pr_docs/45058/format/StatisticsSchema.html It

Re: [DISCUSS] Arrow array representation of statistics

2024-12-17 Thread Felipe Oliveira Carvalho
> I think it's fair not to mention any other Arrow-like transport mechanism > since the benefits of transporting the statistics as an Arrow array are > less clear right now. When we (or applications) start thinking about more advanced statistics like compressed histograms and sketch data structure

Re: [DISCUSS] Arrow array representation of statistics

2024-12-13 Thread Dewey Dunnington
First of all, thank you for driving this proposal! I don't think there's anything particularly bad or wrong about mentioning the C data interface in the title of the document...my initial comment was mostly a reaction to the fact that most of the content of the proposal is describing this schema, w

Re: [DISCUSS] Arrow array representation of statistics

2024-12-11 Thread David Li
I think the feedback is more along the lines of: we can just standardize a representation of statistics, without referencing where it's used (C Data Interface or otherwise). So people are free to use it wherever they want, whether C Data Interface or IPC or somewhere else. At the same time, we a