Re: [C++][CMake] How to add new .cc & .h files?

Hi, You need to add .cc and .h to ARROW_SRCS. For ORC related .cc and .h, https://github.com/apache/arrow/blob/master/cpp/src/arrow/CMakeLists.txt#L466 is the best location to do it. Thanks, -- kou In <7b643229-615d-44d6-856a-98609e4af...@gmail.com> "[C++][CMake] How to add new .cc & .h files?

Re: [Rust] Arrow in WebAssemby

Domink's point is that the IPC reader currently first writes the whole thing into a Vec, and then copies all of that to buffers using IPC::Buffer offsets and lengths. Thus, it performs 2 memcopies of the whole data and needs to hold 2x the required memory (the Vec and the arrow::Buffers). I notice

[C++][CMake] How to add new .cc & .h files?

Hi, Right now I’m working on ARROW-11297 which adds WriterOptions to the ORCWriter (and will be a separate PR). After adding new files (adapter_options.h & adapter_options.cc ) I found that src/arrow/CMakeFiles/arrow_objlib.dir/adapters/orc/adapter_options.cc.o doesn

Re: [Flight Extension] Request for Comments

> > You know what? This is actually a nicer solution than I am giving it > credit for. I've been trying to think about how to handle the > Integer.MAX_VALUE limit that arrow strongly suggests to maintain > compatibility with Java, while still respecting the need to apply an update > atomically. Fo

Re: [DISCUSS] Conventions for values masked by null bits

> > If the underlying values were allocated but not initialized they may leak > private information such as private keys, passwords, or tokens which were > placed in that memory then freed by an application without overwrite > I would not be concerned with the security implications of reading thos

Re: [Flight Extension] Request for Comments

>note that FlightData already has a separate app_metadata field That is an interesting point; are there any conventions on how to use the app_metadata compatibly without stepping on other ideas/projects doing the same? It would be convenient for the server to verify that the client is making the r

Re: [DISCUSS] integration against parquet files?

I had some similar thoughts recently and wrote a bit of code to automate the process of running PySpark jobs that write sample Parquet files with multiple versions of Spark: https://github.com/ursa-labs/parquet-lot That repo provides some lightweight scaffolding and includes two simple example tas

Re: [Rust] Arrow in WebAssemby

Thank you for filing the ticket. I wonder if you mean this reader: https://docs.rs/arrow/3.0.0/arrow/ipc/reader/struct.FileReader.html#method.try_new If so, while it is called a `FileReader` I think that is somewhat misleading. It requires something that implements `std::io::Read` -- which `&[u8]

Re: [Proposal] Improving PR tracking in Github with PR labels

@Weston: Adding a system to figure out when PRs are waiting for review would really help me personally. I spend non trivial time trying to make sure we don't miss any Rust PRs that come in and it gets challenging to filter out the ones that are WIP. Having a bot add some labels would be great. @N

Re: [ANNOUNCE] New Arrow PMC member: Andrew Lamb

Thank you all -- I look forward to working to make the Arrow project even more impactful than it currently is. It is a great community to be a part of On Mon, Mar 8, 2021 at 4:55 PM paddy horan wrote: > Congrats Andrew! > > > > From: Krisztián Szűcs > Sent: Mo

Re: [ANNOUNCE] New Arrow PMC member: Andrew Lamb

Congrats Andrew! From: Krisztián Szűcs Sent: Monday, March 8, 2021 3:14:03 PM To: dev Subject: Re: [ANNOUNCE] New Arrow PMC member: Andrew Lamb Congratulations Andrew! On Mon, Mar 8, 2021 at 7:43 PM Daniël Heres wrote: > > Congrats Andrew, well deserved! > >

Re: [ANNOUNCE] New Arrow PMC member: Jorge Leitão

Congrats Jorge! Well deserved! From: Krisztián Szűcs Sent: Monday, March 8, 2021 3:14 PM To: dev Subject: Re: [ANNOUNCE] New Arrow PMC member: Jorge Leitão Congratulations Jorge! On Mon, Mar 8, 2021 at 7:44 PM Daniël Heres wrote: > > Congrats Jorge, well dese

Re: [DISCUSS] Conventions for values masked by null bits

Yeah I agree that the general sentinel support has lots of challenges, but in the more narrow case of "read this parquet or CSV file and return an R data.frame", the lifetime of the arrays in question is contained. On Mon, Mar 8, 2021 at 12:42 PM Wes McKinney wrote: > It's a bit outside the scop

Re: [Flight Extension] Request for Comments

Hey - pretty much, I think. I'd just like to note that FlightData already has a separate app_metadata field, for metadata on top of any Arrow-level data, so you could ship the Barrage metadata alongside the first record batch, without having to modify anything about the record batch itself, and

Re: [DISCUSS] Conventions for values masked by null bits

It's a bit outside the scope of this discussion, but I've looked at those R Jira issues before, and I think the challenge is how the code will "know" what fill values are being used. If you start putting field-level metadata in a schema object, you're playing a dangerous game if that schema gets at

Re: [ANNOUNCE] New Arrow PMC member: Jorge Leitão

Congratulations Jorge! On Mon, Mar 8, 2021 at 7:44 PM Daniël Heres wrote: > > Congrats Jorge, well deserved! > > Op ma 8 mrt. 2021 om 19:25 schreef Fernando Herrera < > fernando.j.herr...@gmail.com>: > > > Congrats Jorge > > > > On Mon, 8 Mar 2021, 17:26 Micah Kornfield, wrote: > > > > > Congrat

Re: [ANNOUNCE] New Arrow PMC member: Andrew Lamb

Congratulations Andrew! On Mon, Mar 8, 2021 at 7:43 PM Daniël Heres wrote: > > Congrats Andrew, well deserved! > > Op ma 8 mrt. 2021 om 19:25 schreef Fernando Herrera < > fernando.j.herr...@gmail.com>: > > > Congrats Andrew > > > > On Mon, 8 Mar 2021, 17:26 Micah Kornfield, wrote: > > > > > Cong

Re: [ANNOUNCE] New Arrow PMC member: Andrew Lamb

Congratulations Andrew! It is being a pleasure working with you!!! :D Best, Jorge On Mon, Mar 8, 2021 at 6:23 PM Wes McKinney wrote: > The Project Management Committee (PMC) for Apache Arrow has invited > Andrew Lamb to become a PMC member and we are pleased to announce > that Andrew has accep

Re: [ANNOUNCE] New Arrow PMC member: Jorge Leitão

Congrats! Neal On Mon, Mar 8, 2021 at 11:05 AM Sven Wagner-Boysen < sven.wagnerboy...@gmail.com> wrote: > Congratulations Jorge! > > On Mon, Mar 8, 2021 at 7:44 PM Daniël Heres wrote: > > > Congrats Jorge, well deserved! > > > > Op ma 8 mrt. 2021 om 19:25 schreef Fernando Herrera < > > fernando

Re: [ANNOUNCE] New Arrow PMC member: Andrew Lamb

Congrats! Neal On Mon, Mar 8, 2021 at 11:05 AM Sven Wagner-Boysen < sven.wagnerboy...@gmail.com> wrote: > Congratulations Andrew! > > On Mon, Mar 8, 2021 at 7:43 PM Daniël Heres wrote: > > > Congrats Andrew, well deserved! > > > > Op ma 8 mrt. 2021 om 19:25 schreef Fernando Herrera < > > fernan

Re: [DISCUSS] Conventions for values masked by null bits

What was the resolution of this discussion? Was a JIRA made? It occurred to me recently that, if we decided that values masked by null bits need to be filled with a known value, this could open up optimizations in some use cases. For example, when reading a file into R, if we could specify what to

Re: [ANNOUNCE] New Arrow PMC member: Jorge Leitão

Congratulations Jorge! On Mon, Mar 8, 2021 at 7:44 PM Daniël Heres wrote: > Congrats Jorge, well deserved! > > Op ma 8 mrt. 2021 om 19:25 schreef Fernando Herrera < > fernando.j.herr...@gmail.com>: > > > Congrats Jorge > > > > On Mon, 8 Mar 2021, 17:26 Micah Kornfield, > wrote: > > > > > Congra

Re: [ANNOUNCE] New Arrow PMC member: Andrew Lamb

Congratulations Andrew! On Mon, Mar 8, 2021 at 7:43 PM Daniël Heres wrote: > Congrats Andrew, well deserved! > > Op ma 8 mrt. 2021 om 19:25 schreef Fernando Herrera < > fernando.j.herr...@gmail.com>: > > > Congrats Andrew > > > > On Mon, 8 Mar 2021, 17:26 Micah Kornfield, > wrote: > > > > > Con

Re: [ANNOUNCE] New Arrow PMC member: Jorge Leitão

Congrats Jorge, well deserved! Op ma 8 mrt. 2021 om 19:25 schreef Fernando Herrera < fernando.j.herr...@gmail.com>: > Congrats Jorge > > On Mon, 8 Mar 2021, 17:26 Micah Kornfield, wrote: > > > Congratulations Jorge! > > > > On Mon, Mar 8, 2021 at 9:25 AM Wes McKinney wrote: > > > > > The Projec

Re: [ANNOUNCE] New Arrow PMC member: Andrew Lamb

Congrats Andrew, well deserved! Op ma 8 mrt. 2021 om 19:25 schreef Fernando Herrera < fernando.j.herr...@gmail.com>: > Congrats Andrew > > On Mon, 8 Mar 2021, 17:26 Micah Kornfield, wrote: > > > Congratulations Andrew! > > > > On Mon, Mar 8, 2021 at 9:23 AM Wes McKinney wrote: > > > > > The Pro

Re: [ANNOUNCE] New Arrow PMC member: Jorge Leitão

Congrats Jorge On Mon, 8 Mar 2021, 17:26 Micah Kornfield, wrote: > Congratulations Jorge! > > On Mon, Mar 8, 2021 at 9:25 AM Wes McKinney wrote: > > > The Project Management Committee (PMC) for Apache Arrow has invited > > Jorge Leitão to become a PMC member and we are pleased to announce > > t

Re: [ANNOUNCE] New Arrow PMC member: Andrew Lamb

Congrats Andrew On Mon, 8 Mar 2021, 17:26 Micah Kornfield, wrote: > Congratulations Andrew! > > On Mon, Mar 8, 2021 at 9:23 AM Wes McKinney wrote: > > > The Project Management Committee (PMC) for Apache Arrow has invited > > Andrew Lamb to become a PMC member and we are pleased to announce > >

Re: [ANNOUNCE] New Arrow PMC member: Jorge Leitão

Congratulations Jorge! On Mon, Mar 8, 2021 at 9:25 AM Wes McKinney wrote: > The Project Management Committee (PMC) for Apache Arrow has invited > Jorge Leitão to become a PMC member and we are pleased to announce > that Jorge has accepted. > > Congratulations and welcome! >

Re: [ANNOUNCE] New Arrow PMC member: Andrew Lamb

Congratulations Andrew! On Mon, Mar 8, 2021 at 9:23 AM Wes McKinney wrote: > The Project Management Committee (PMC) for Apache Arrow has invited > Andrew Lamb to become a PMC member and we are pleased to announce > that Andrew has accepted. > > Congratulations and welcome! >

[ANNOUNCE] New Arrow PMC member: Jorge Leitão

The Project Management Committee (PMC) for Apache Arrow has invited Jorge Leitão to become a PMC member and we are pleased to announce that Jorge has accepted. Congratulations and welcome!

[ANNOUNCE] New Arrow PMC member: Andrew Lamb

The Project Management Committee (PMC) for Apache Arrow has invited Andrew Lamb to become a PMC member and we are pleased to announce that Andrew has accepted. Congratulations and welcome!

[NIGHTLY] Arrow Build Report for Job nightly-2021-03-08-0

Arrow Build Report for Job nightly-2021-03-08-0 All tasks: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-08-0 Failed Tasks: - conda-linux-gcc-py39-aarch64: URL: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-08-0-drone-conda-linux