Re: [VOTE][RUST] Release Apache Arrow Rust Object Store 0.12.0 RC2

t;> https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-object-store-rs-0.12.0-rc2 >> [3]: >> https://github.com/apache/arrow-rs/blob/3da5e0d010cb605c2f180c95643fc57dedf4f0fb/object_store/CHANGELOG.md >> [4]: >> https://github.com/apache/arrow-rs/blob/master/object_store/dev/release/verify-release-candidate.sh -- Xuanwo https://xuanwo.io/

Re: [VOTE][RUST] Release Apache Arrow Rust Object Store 0.12.0 RC1

ase. There is a script [4] that automates some of >> >> the verification. >> >> >> >> The vote will be open for at least 72 hours. >> >> >> >> [ ] +1 Release this as Apache Arrow Rust Object Store >> >> [ ] +0 >> >> [ ] -1 Do not release this as Apache Arrow Rust Object Store because... >> >> >> >> [1]: >> >> https://github.com/apache/arrow-rs/tree/89a2ef8e06088c29433c41a8d8f6f2a46ba8f399 >> >> [2]: >> >> https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-object-store-rs-0.12.0-rc1 >> >> [3]: >> >> https://github.com/apache/arrow-rs/blob/89a2ef8e06088c29433c41a8d8f6f2a46ba8f399/object_store/CHANGELOG.md >> >> [4]: >> >> https://github.com/apache/arrow-rs/blob/main/object_store/dev/release/verify-release-candidate.sh >> >> -- Xuanwo https://xuanwo.io/

Re: [VOTE][RUST] Release Apache Arrow Rust 53.4.1 RC1

e/962558546b9f441b5b7d039624d068c36a3ef69e >> > [2]: >> https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-rs-53.4.1-rc1 >> > [3]: >> > >> https://github.com/apache/arrow-rs/blob/962558546b9f441b5b7d039624d068c36a3ef69e/CHANGELOG.md >> > [4]: >> > >> https://github.com/apache/arrow-rs/blob/master/dev/release/verify-release-candidate.sh >> > [5]: https://github.com/apache/arrow-rs/issues/7232 >> > [6]: https://lists.apache.org/thread/yjcjkv79d3wkpoj5d41y9q8ozvld3kxl >> -- Xuanwo https://xuanwo.io/

Re: [VOTE][RUST] Release Apache Arrow Rust 54.2.1 RC1

; [1]: >> https://github.com/apache/arrow-rs/tree/3f564688cbcd8351e18b35c6286c97f5dd0a8606 >> [2]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-rs-54.2.1-rc1 >> [3]: >> https://github.com/apache/arrow-rs/blob/3f564688cbcd8351e18b35c6286c97f5dd0a8606/CHANGELOG.md >> [4]: >> https://github.com/apache/arrow-rs/blob/main/dev/release/verify-release-candidate.sh >> [5]: https://github.com/apache/arrow-rs/issues/7209 >> -- Xuanwo https://xuanwo.io/

Re: [DISCUSS] Variant Spec Location

ut I don't feel like that should be the initial >> step. >> > > > >>>>>>>>> >> > > > >>>>>>>>> No one is excited about the possibility that the physical >> > > > representations end up diverging, but it feels like we're setting >> > > ourselves >> > > > up for that exact scenario. >> > > > >>>>>>>>> >> > > > >>>>>>>>> -Dan >> > > > >>>>>>>>> >> > > > >>>>>>>>> >> > > > >>>>>>>>> On Wed, Aug 14, 2024 at 6:54 AM Fokko Driesprong < >> > > > fo...@apache.org> wrote: >> > > > >>>>>>>>>> >> > > > >>>>>>>>>> +1 to what's already being said here. It is good to copy >> the >> > > > spec to Iceberg and add context that's specific to Iceberg, but at >> the >> > > same >> > > > time, we should maintain compatibility. >> > > > >>>>>>>>>> >> > > > >>>>>>>>>> Kind regards, >> > > > >>>>>>>>>> Fokko >> > > > >>>>>>>>>> >> > > > >>>>>>>>>> Op wo 14 aug 2024 om 15:30 schreef Manu Zhang < >> > > > owenzhang1...@gmail.com>: >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> +1 to copy the spec into our repository. I think the best >> > way >> > > > to keep compatibility is building integration tests. >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> Thanks, >> > > > >>>>>>>>>>> Manu >> > > > >>>>>>>>>>> >> > > > >>>>>>>>>>> On Wed, Aug 14, 2024 at 8:27 PM Péter Váry < >> > > > peter.vary.apa...@gmail.com> wrote: >> > > > >>>>>>>>>>>> >> > > > >>>>>>>>>>>> Thanks Russell and Aihua for pushing Variant support! >> > > > >>>>>>>>>>>> >> > > > >>>>>>>>>>>> Given the differences between the supported types and >> the >> > > > lack of interest from the other project, I think it is reasonable to >> > > > duplicate the specification to our repository. >> > > > >>>>>>>>>>>> I would give very strong emphasis on sticking to the >> Spark >> > > > spec as much as possible, to keep compatibility as much as possible. >> > > Maybe >> > > > even revert to a shared specification if the situation changes. >> > > > >>>>>>>>>>>> >> > > > >>>>>>>>>>>> Thanks, >> > > > >>>>>>>>>>>> Peter >> > > > >>>>>>>>>>>> >> > > > >>>>>>>>>>>> Aihua Xu ezt írta (időpont: 2024. >> > aug. >> > > > 13., K, 19:52): >> > > > >>>>>>>>>>>>> >> > > > >>>>>>>>>>>>> Thanks Russell for bringing this up. >> > > > >>>>>>>>>>>>> >> > > > >>>>>>>>>>>>> This is the main blocker to move forward with the >> Variant >> > > > support in Iceberg and hopefully we can have a consensus. To me, I >> also >> > > > feel it makes more sense to move the spec into Iceberg rather than >> > Spark >> > > > engine owns it and we try to keep it compatible with Spark spec. >> > > > >>>>>>>>>>>>> >> > > > >>>>>>>>>>>>> Thanks, >> > > > >>>>>>>>>>>>> Aihua >> > > > >>>>>>>>>>>>> >> > > > >>>>>>>>>>>>> On Mon, Aug 12, 2024 at 6:50 PM Russell Spitzer < >> > > > russell.spit...@gmail.com> wrote: >> > > > >>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>> Hi Y’all, >> > > > >>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>> We’ve hit a bit of a roadblock with the Variant >> > Proposal, >> > > > while we were hoping to move the Variant and Shredding specifications >> > > from >> > > > Spark into Iceberg there doesn’t seem to be a lot of interest in >> that. >> > > > Unfortunately, I think we have a number of issues with just linking >> to >> > > the >> > > > Spark project directly from within Iceberg and I believe we need to >> > copy >> > > > the specifications into our repository. >> > > > >>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>> There are a few reasons why i think this is necessary >> > > > >>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>> First, we have a divergence of types already. The >> Spark >> > > > Specification already includes types which Iceberg has no definition >> > for >> > > > (19, 20 - Interval Types) and Iceberg already has a type which is not >> > > > included within the Spark Specification (Time) and will soon have >> more >> > > with >> > > > TimestampNS, and Geo. >> > > > >>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>> Second, We would like to make sure that Spark is not a >> > > hard >> > > > dependency for other engines. We are working with several >> implementers >> > of >> > > > the Iceberg spec and it has previously been agreed that it would be >> > best >> > > if >> > > > the source of truth for Variant existed in an engine and file format >> > > > neutral location. The Iceberg project has a good open model of >> > governance >> > > > and, as we have seen so far discussing Variant, open and active >> > > > collaboration. This would also help as we can strictly version our >> > > changes >> > > > in-line with the rest of the Iceberg spec. >> > > > >>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>> Third, The Shredding spec is not quite finished and >> > > > requires some group analysis and discussion before we commit it. I >> > think >> > > > again the Iceberg community is probably the right place for this to >> > > happen >> > > > as we have already started discussions here on these topics. >> > > > >>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>> For these reasons I think we should go with a direct >> > copy >> > > > of the existing specification from the Spark Project and move ahead >> > with >> > > > our discussions and modifications within Iceberg. That said, I do not >> > > want >> > > > to diverge if possible from the Spark proposal. For example, although >> > we >> > > do >> > > > not use the Interval types above, I think we should not reuse those >> > type >> > > > ids within our spec. Iceberg's Variant Spec types 19 and 20 would >> > remain >> > > > unused along with any other types we think are not applicable. We >> > should >> > > > strive whenever possible to allow for compatibility. >> > > > >>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>> In the interest of moving forward with this proposal I >> > am >> > > > hoping to see if anyone in the community objects to this plan going >> > > forward >> > > > or has a better alternative. >> > > > >>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>> As always I am thankful for your time and am eager to >> > hear >> > > > back from everyone, >> > > > >>>>>>>>>>>>>> Russ >> > > > >>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>> >> > > > >> > > >> > >> -- Xuanwo https://xuanwo.io/

Re: [VOTE][RUST] Release Apache Arrow Rust Object Store 0.11.0 RC2

Lamb wrote: > Hi, > > I would like to propose a release of Apache Arrow Rust Object Store > Implementation, version 0.11.0. > > NOTE this is RC2 (RC1 had a problem[5]). Thank you Xuanwo for reporting the > issue and helping to fix it. > > This

Re: [VOTE][RUST] Release Apache Arrow Rust Object Store 0.11.0 RC1

Great, I have seen that. Thanks for the quick fix. On Tue, Aug 13, 2024, at 18:01, Andrew Lamb wrote: > Thank you Xuanwo for the report > > I have filed an issue[1] to address this and will create a new RC / voting > thread > > Andrew > > [1] https://github.com/apache/arr

Re: [VOTE][RUST] Release Apache Arrow Rust Object Store 0.11.0 RC1

che/arrow-rs/blob/fe03d393933493b6fa59cd1d05963a0c17c47831/object_store/CHANGELOG.md > [4]: > https://github.com/apache/arrow-rs/blob/master/object_store/dev/release/verify-release-candidate.sh -- Xuanwo https://xuanwo.io/

Re: [DISCUSS][RUST] Making a separate repository for Rust `object_store` crate

et if you would like to participate >> in the discussion. >> >> Thanks, >> Andrew >> >> [1]: https://github.com/apache/arrow-rs/issues/6183 >> -- Xuanwo https://xuanwo.io/

Re: [DISCUSS] Split Go release process

ot;import" to > "github.com/apache/arrow-go/arrow" from > "github.com/apache/arrow/go/arrow" > > > What do you think about this? > > > Thanks, > -- > kou -- Xuanwo https://xuanwo.io/

Re: [DISCUSS] Donation of a User-Defined Function Framework for Apache Arrow

costs to > downstream crates. > > Andrew > > [1]: > https://docs.rs/arrow/latest/arrow/array/struct.PrimitiveArray.html#method.unary_mut > > On Sat, Jun 29, 2024 at 1:47 AM Xuanwo wrote: > >> > That said, wherever it ends up, there should be the agreement of &g

Re: [DISCUSS] Donation of a User-Defined Function Framework for Apache Arrow

would be good for it to be part of the community, but only if it's not > going to end up just bitrotting somewhere. > > --Matt > > On Fri, Jun 28, 2024, 8:49 PM Xuanwo wrote: > >> Hi, >> >> This UDF implementation doesn’t depend on DataFusion. It can work with an

Re: [DISCUSS] Donation of a User-Defined Function Framework for Apache Arrow

arrow/latest/arrow/compute/fn.unary.html> or binary < >>>> https://docs.rs/arrow/latest/arrow/compute/fn.binary.html> kernels, >>>> which potentially allows for vectorization. >>>>> >>>>> Both examples you showed are not vectorized. The

[DISCUSS] Donation of a User-Defined Function Framework for Apache Arrow

Hello, everyone. I start this thread to disscuss the donation of a User-Defined Function Framework for Apache Arrow. Feel free to review and leave your comments here. For live review, please visit: https://hackmd.io/@xuanwo/apache-arrow-udf The original content also pasted here for a quick