I see your point is very valid, in my use case the consumer which will be
using an SDK we are exposing an api:
The method reads FlightStream in complete with a tunable batch size, the
iterator will not be fully traversed only if the process crashes, or
explicitly the user stops the process.
def
Yes, the first example was returning an iterator to the caller which would
interact with the stream after the client is closed (lazy evaluation). The
new code should work but may cause a leak if the iterator is not fully
traversed. You may want to return a type which has both the Iterator and
AutoC
I believe I figured it out i changed slightly the logic, post the changes
the issue is not prevalent..
def fetchDataStream(details: ObjectStoreDetails): Iterator[FlightStream] = {
logger.info(s"Fetching data for S3 path: ${details.s3Path}")
val ticketStr = buildTicketStr(details)
logger.info
Haven't tested the code but isn't `FlightStream` a closeable as well?
On Fri, Oct 25, 2024 at 3:40 AM Susmit Sarkar
wrote:
> Hi Team,
>
> We are seeing the issue often with Memory Leak:
>
> *JDK 11*
>
> "org.apache.arrow" % "arrow-flight" % "17.0.0",
> "org.apache.arrow" % "arrow-vector" % "17.0
Hi Team,
We are seeing the issue often with Memory Leak:
*JDK 11*
"org.apache.arrow" % "arrow-flight" % "17.0.0",
"org.apache.arrow" % "arrow-vector" % "17.0.0",
"org.apache.arrow" % "flight-core" % "17.0.0",
4-10-25 15:25:06.394 [main] ERROR o.apache.arrow.memory.BaseAllocator -
Memory was le
