Re: [VOTE][RUST] Release Apache Arrow Rust 19.0.0 RC1

2022-07-22 Thread QP Hou
+1 (binding) On Fri, Jul 22, 2022 at 5:01 PM L. C. Hsieh wrote: > > +1 (non-binding) > > Tested on Intel Macbook. > > Thanks Andrew! > > On Fri, Jul 22, 2022 at 12:34 PM Ian Joiner wrote: > > > > +1 (Non-binding) > > > > Tested on a MacBook Pro with macOS 12.2.1 & Apple M1 Pro chip > > > > On Fr

Re: [VOTE][RUST] Release Apache Arrow Rust 19.0.0 RC1

2022-07-22 Thread L. C. Hsieh
+1 (non-binding) Tested on Intel Macbook. Thanks Andrew! On Fri, Jul 22, 2022 at 12:34 PM Ian Joiner wrote: > > +1 (Non-binding) > > Tested on a MacBook Pro with macOS 12.2.1 & Apple M1 Pro chip > > On Fri, Jul 22, 2022 at 12:56 PM Andrew Lamb wrote: > > > Hi, > > > > I would like to propose a

Re: [C++] ResumeProducing Future Causing Blocking

2022-07-22 Thread Weston Pace
> Li gave a quick suggestion with blocking queues which I'm using now, though I > heard that approach is not suitable for production due to thread blocking. I think it's a reasonable workaround if you have a relatively small number of sources. > 1) I'm assuming here that L(N) and R(N) just repre

Re: [Rust] IPC Format / Feather support in Datafusion

2022-07-22 Thread Will Jones
FYI It looks like there is active work to change the Python [1] and R [2] implementations (built on C++) to write .arrow instead of .feather. [1] https://issues.apache.org/jira/browse/ARROW-17089 [2] https://issues.apache.org/jira/browse/ARROW-17088 On Fri, Jul 22, 2022 at 11:16 AM Aldrin wrote:

RE: [C++] ResumeProducing Future Causing Blocking

2022-07-22 Thread Ivan Chau
Thanks Weston -- the first diagram indeed makes it look like backpressure will be a headache. Li gave a quick suggestion with blocking queues which I'm using now, though I heard that approach is not suitable for production due to thread blocking. Some questions about your event loop diagram (th

Re: [C++] ResumeProducing Future Causing Blocking

2022-07-22 Thread Weston Pace
Probably attachments don't work with the ML. Here are links to my drive: Current Serial: https://drive.google.com/file/d/1URbDAqXVSYfHJixzHA00CI1f0aedms8W/view?usp=sharing Proposed Serial: https://drive.google.com/file/d/1JpQiIVaGAL9mrderkid5uf888zGx21fO/view?usp=sharing On Fri, Jul 22, 2022 at

Re: [VOTE][RUST] Release Apache Arrow Rust 19.0.0 RC1

2022-07-22 Thread Ian Joiner
+1 (Non-binding) Tested on a MacBook Pro with macOS 12.2.1 & Apple M1 Pro chip On Fri, Jul 22, 2022 at 12:56 PM Andrew Lamb wrote: > Hi, > > I would like to propose a release of Apache Arrow Rust Implementation, > version 19.0.0. > > This release candidate is based on commit: > c3e019f3011a902a

RE: [C++] ResumeProducing Future Causing Blocking

2022-07-22 Thread Ivan Chau
Hi Weston, Not sure if the diagrams came through here -- is there some other place I need to view them? Ivan -Original Message- From: Weston Pace Sent: Thursday, July 21, 2022 10:59 PM To: dev@arrow.apache.org Subject: Re: [C++] ResumeProducing Future Causing Blocking > Do you have an

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-22 Thread Arkadiy Vertleyb (BLOOMBERG/ 120 PARK)
Or live with the warnings. Or cast his sizes to _int64, if the warnings are critical. From: dev@arrow.apache.org At: 07/22/22 14:06:34 UTC-4:00To: Arkadiy Vertleyb (BLOOMBERG/ 120 PARK ) , dev@arrow.apache.org Subject: Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win

Re: [C++] Control flow and scheduling in C++ Engine operators / exec nodes

2022-07-22 Thread Li Jin
Hi! Since the scheduler improvement work came up in some recent discussions about how backpresures are handled in Acero, I am curious if there has been any more progress on this since May or any future plans? Thanks, Li On Mon, May 23, 2022 at 10:37 PM Weston Pace wrote: > > About point 2. I h

Re: [Rust] IPC Format / Feather support in Datafusion

2022-07-22 Thread Aldrin
sorry, I meant "...especially *for* the rust community if they are just using IPC directly for file formats." Aldrin Montana Computer Science PhD Student UC Santa Cruz On Fri, Jul 22, 2022 at 11:14 AM Aldrin wrote: > I always assumed IPC was when it was in memory, feather was when it was > per

Re: [Rust] IPC Format / Feather support in Datafusion

2022-07-22 Thread Aldrin
I always assumed IPC was when it was in memory, feather was when it was persisted as files. That distinction/indirection may be totally moot, though, especially if the rust community is just using IPC directly for file formats. In my mind, feather V1 came just before IPC and started using IPC sinc

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-22 Thread James Duong
The problem with (2) is that a library user can only work around it by suppressing warnings with pragmas when including Arrow headers. This is somewhat unfriendly from a user perspective. On Fri., Jul. 22, 2022, 08:11 Arkadiy Vertleyb (BLOOMBERG/ 120 PARK), < avertl...@bloomberg.net> wrote: > Th

Re: [Rust] IPC Format / Feather support in Datafusion

2022-07-22 Thread Andrew Lamb
For what it is worth, in the Rust Arrow community, I have only ever heard the format referred to as "IPC Format" , we haven't used the term Feather that I know of. On Fri, Jul 22, 2022 at 12:41 PM David Li wrote: > FeatherV2 is the same thing as the IPC file format ( > https://arrow.apache.org/f

[VOTE][RUST] Release Apache Arrow Rust 19.0.0 RC1

2022-07-22 Thread Andrew Lamb
Hi, I would like to propose a release of Apache Arrow Rust Implementation, version 19.0.0. This release candidate is based on commit: c3e019f3011a902a344758969b1cfc3604f3c2d7 [1] The proposed release tarball and signatures are hosted at [2]. The changelog is located at [3]. Please download, ve

Re: [Rust] IPC Format / Feather support in Datafusion

2022-07-22 Thread David Li
FeatherV2 is the same thing as the IPC file format (https://arrow.apache.org/faq/). This confusion keeps coming up, so: how do we want to brand Arrow IPC? As "Feather", or as "Arrow IPC Files"? It seems the latter, but even right now Arrow Dataset still writes files with .feather extension (tho

Re: [Rust] IPC Format / Feather support in Datafusion

2022-07-22 Thread Andrew Lamb
I am not familiar with feather -- there are some Draft PRs to add support for reading [1] and writing [2] the IPC file format which may be related to your question [1] https://github.com/apache/arrow-datafusion/pull/1858 [2] https://github.com/apache/arrow-datafusion/pull/1893 On Fri, Jul 22, 20

[RESULT] Re: [VOTE] Accept donation of Rust Object Store donation

2022-07-22 Thread Andrew Lamb
With 6 +1 votes (4 binding) the donation is accepted. Thank you all for voting and your patience. I will proceed with the plan on [1] Andrew [1] https://github.com/apache/arrow-rs/issues/2030 On Tue, Jul 19, 2022 at 4:25 AM Daniël Heres wrote: > +1 (binding) > > On Tue, Jul 19, 2022, 09:03 M

[Rust] IPC Format / Feather support in Datafusion

2022-07-22 Thread Ian Joiner
Hi, I'd like to ask about the level of Feather support in Datafusion. Can we read and write Feather files? I don't really find any information while searching through the repo or checking the docs. Thanks, Ian

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-22 Thread Arkadiy Vertleyb (BLOOMBERG/ 120 PARK)
This is what I would do: 1) Suppress warnings in cmake input file for this particular platform. This will let the user build the arrow libs. 2) Let those users who care about 32 bits worry about the warnings in the headers. Regards, Arkadiy From: dev@arrow.apache.org At: 07/22/22 10:18:09 UT

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-22 Thread Antoine Pitrou
Ah, that's a good point. Then we should probably use explicit casts. Le 22/07/2022 à 15:58, James Duong a écrit : Seems reasonable to suppress this warning on this single platform. However if we do this using a compiler option, we may hide warnings in public headers from the build. This isn't

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-22 Thread James Duong
Seems reasonable to suppress this warning on this single platform. However if we do this using a compiler option, we may hide warnings in public headers from the build. This isn't great since library users may have policies that disallow warnings. On Fri., Jul. 22, 2022, 05:47 Antoine Pitrou, wro

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-22 Thread Antoine Pitrou
We could perhaps suppress the integer downcast warnings, but only on 32-bit Windows (not 64-bit, not other platforms). Regards Antoine. Le 22/07/2022 à 14:42, Arkadiy Vertleyb (BLOOMBERG/ 120 PARK) a écrit : Hi James. I don't have strong feelings about whose PR is used and how exactly th

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-22 Thread Arkadiy Vertleyb (BLOOMBERG/ 120 PARK)
Hi James. I don't have strong feelings about whose PR is used and how exactly the issue is fixed. All I care about at this point is working (and maintained) MSVC 32 bit version. I need this for my project at work. If you feel that your PR will solve this issue, and is close to being approved,

Re: Preparing for version 9.0.0 release

2022-07-22 Thread Jacob Wujciak
Hello Everyone, Currently there have been 452 JIRA tickets resolved for 9.0.0 with 23 tickets (ex. blockers) still open, good job everyone! Thanks to everyone's work we were able to reduce the number of blockers and have PRs [1][2][3] for 3/5 open blockers that mostly require reviews. The followi