Re: [C++] ResumeProducing Future Causing Blocking

g if the scenario would be > avoided in the Event Loop implementation. > > Ivan > > > -----Original Message- > From: Weston Pace > Sent: Friday, July 22, 2022 3:40 PM > To: dev@arrow.apache.org > Subject: Re: [C++] ResumeProducing Future Causing Blocking > > Pr

RE: [C++] ResumeProducing Future Causing Blocking

eston Pace Sent: Friday, July 22, 2022 3:40 PM To: dev@arrow.apache.org Subject: Re: [C++] ResumeProducing Future Causing Blocking Probably attachments don't work with the ML. Here are links to my drive: Current Serial: https://drive.google.com/file/d/1URbDAqXVSYfHJixzHA00CI1f0aedms8W/v

Re: [C++] ResumeProducing Future Causing Blocking

doesn't seem like a great idea; It > > gets pretty hairy when things are getting deconstructed at the end of > > execution (it seems like ResumeProducing still has some invalid reads > > despite checking finished() on the source nodes), and I'm assuming we don't > > wan

RE: [C++] ResumeProducing Future Causing Blocking

Hi Weston, Not sure if the diagrams came through here -- is there some other place I need to view them? Ivan -Original Message- From: Weston Pace Sent: Thursday, July 21, 2022 10:59 PM To: dev@arrow.apache.org Subject: Re: [C++] ResumeProducing Future Causing Blocking > Do you h

Re: [C++] ResumeProducing Future Causing Blocking

es), and I'm assuming we don't want to spin up more threads > anyway. I also looked into the plan's executor (ScheduleTask, etc.), but I > believe this waits for the task to complete, so it causes blocking in the > processing. > > Do you have any suggestions for a tem

RE: [C++] ResumeProducing Future Causing Blocking

depend on batch ordering/completion rather than memory size. Ivan -Original Message- From: Weston Pace Sent: Wednesday, July 20, 2022 8:31 PM To: dev@arrow.apache.org Subject: Re: [C++] ResumeProducing Future Causing Blocking > 4) control is not returned to the processing thread Ye

RE: [C++] ResumeProducing Future Causing Blocking

etion rather than memory size. Ivan -Original Message- From: Weston Pace Sent: Wednesday, July 20, 2022 8:31 PM To: dev@arrow.apache.org Subject: Re: [C++] ResumeProducing Future Causing Blocking > 4) control is not returned to the processing thread Yes, it looks like the current implem

Re: [C++] ResumeProducing Future Causing Blocking

> 4) control is not returned to the processing thread Yes, it looks like the current implementation does not return control to the processing thread, but I think this is correct, or at least "as designed". The thread will be used to continue iterating the source. > control is not returned to the

Re: [C++] ResumeProducing Future Causing Blocking

Hi, Futures run callbacks on the thread that marks then as finished. It seems that inside of the Source node’s generator loop does add a callback (https://github.com/iChauster/arrow/blob/asof_join2/cpp/src/arrow/compute/exec/source_node.cc#L130

[C++] ResumeProducing Future Causing Blocking

Hi all, I am currently working on writing a manual back-pressure mechanism for AsOfJoin. We are trying a simple version where we maintain a buffer of batches from one of the input sources (our left table source). We want to pause production when we reach a certain number of batches, and resume pro