Re: SplittableDoFn-based source doesn't efficiently scale up in Dataflow

2022-05-09 Thread Claire McGinty
Can you clarify a bit what you mean by being over-aggressive in the splitRestriction? We can't go any smaller as far as the unit of splittability (a single row group). Thanks! -Claire On Tue, May 3, 2022 at 9:14 PM Robert Bradshaw wrote: > On Tue, May 3, 2022 at 10:39 AM Claire McGinty > wrote

Re: IllegalMutationException in PTransform

2022-05-09 Thread Yuri Jin
Hi Reuven. I have overridden and implemented equals() and got the same exception. With more testing, I suspect the part of parsing the byte array into a http request inside inputAdaptor.adapt() . We are using com.athaydes.rawhttp.rawhttp-core. https://github.com/renatoathaydes/rawhttp/blob/core-2