Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-11 Thread Gabor Somogyi
Well, the easiest is to cherry-pick to the top of the latest distro what you build. When not possible then local snapshot build + use it in local execution can be an option. G On Mon, Dec 9, 2024 at 6:17 PM Salva Alcántara wrote: > Hey Gabor! What would be the easiest way for me to test the cha

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-09 Thread Salva Alcántara
Hey Gabor! What would be the easiest way for me to test the change locally? I guess now that it has been merged I could simply bump to the current snapshot version right? But what is the common approach for testing a change in a local project before it gets approved/merged? BTW I was importing the

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-08 Thread Salva Alcántara
{ "emoji": "👍", "version": 1 }

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-08 Thread Gabor Somogyi
It would be good to report back whether it's solved your issue, but since we've added automated test + I've tested it locally it's not urgent. On Sun, Dec 8, 2024 at 8:38 AM Salva Alcántara wrote: > Gabor, I took a look at your PR and it looks good to me but didn't test > your changes locally. I

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-07 Thread Salva Alcántara
Gabor, I took a look at your PR and it looks good to me but didn't test your changes locally. If still necessary, let me know and I'll find the time! On Fri, Dec 6, 2024 at 1:49 PM Gabor Somogyi wrote: > Salva, I've tested the code by myslelf but do you have the possibility to > test this[1] fi

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-06 Thread Salva Alcántara
{ "emoji": "👍", "version": 1 }

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-06 Thread Gabor Somogyi
Salva, I've tested the code by myslelf but do you have the possibility to test this[1] fix? [1] https://github.com/apache/flink/pull/25755 G On Thu, Dec 5, 2024 at 3:15 PM Gabor Somogyi wrote: > I'm intended to file a jira + PR. > > G > > > On Thu, Dec 5, 2024 at 2:40 PM Salva Alcántara > wr

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-05 Thread Salva Alcántara
{ "emoji": "👍", "version": 1 }

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-05 Thread Gabor Somogyi
I'm intended to file a jira + PR. G On Thu, Dec 5, 2024 at 2:40 PM Salva Alcántara wrote: > So what's next? Do you want me to do something Gabor? > > Regards, > > Salva > > On Thu, Dec 5, 2024, 13:48 Gabor Somogyi > wrote: > >> Based on this I can confirm that the maxBatchSize and socketTimeo

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-05 Thread Salva Alcántara
So what's next? Do you want me to do something Gabor? Regards, Salva On Thu, Dec 5, 2024, 13:48 Gabor Somogyi wrote: > Based on this I can confirm that the maxBatchSize and socketTimeout values > are coming from default value Flink config is not considered. > > G > > > On Thu, Dec 5, 2024 at 8

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-05 Thread Salva Alcántara
{ "emoji": "👍", "version": 1 }

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-05 Thread Gabor Somogyi
Based on this I can confirm that the maxBatchSize and socketTimeout values are coming from default value Flink config is not considered. G On Thu, Dec 5, 2024 at 8:20 AM Salva Alcántara wrote: > Sorry for my late reply Gabor, here you are the whole trace: > > SLF4J(W): No SLF4J providers were

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-12-04 Thread Salva Alcántara
Sorry for my late reply Gabor, here you are the whole trace: SLF4J(W): No SLF4J providers were found. SLF4J(W): Defaulting to no-operation (NOP) logger implementation SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details. SLF4J(W): Class path contains SLF4J bindings target

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-07-29 Thread Gabor Somogyi
I've double checked and I think that CollectSinkOperatorFactory is initialized in DataStream.collectAsync without MAX_BATCH_SIZE and SOCKET_TIMEOUT values coming from the Flink config. Could you plz share the whole stacktrace to double check my assumption? G On Tue, Jul 23, 2024 at 12:46 PM Salv

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-07-23 Thread Salva Alcántara
Hi all, Just to share my findings so far. Regarding tweaking the setting, it has been impossible for me to do so. So, the only way to work around this has been to duplicate some Flink code directly to allow me to do the tweak. More precisely, this is how my code looks like now (kudos to my dear co

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-07-22 Thread Salva Alcántara
The same happens with this slight variation: ``` Configuration config = new Configuration(); config.setString("collect-sink.batch-size.max", "100mb"); StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.configure(config); SavepointReader savepoint = Savepoint

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-07-22 Thread Salva Alcántara
Hi Zhanghao, Thanks for your suggestion. Unfortunately, this does not work, I still get the same error message: ``` Record size is too large for CollectSinkFunction. Record size is 9623137 bytes, but max bytes per batch is only 2097152 bytes. Please consider increasing max bytes per batch value b

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-07-21 Thread Zhanghao Chen
Hi, you could increase it as follows: Configuration config = new Configuration(); config.setString(collect-sink.batch-size.max, "10mb"); StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config); From: Salva Alcántara Sent: Satur