That's correct! I only commit when batch has been fully processed and
confirmed successful from external source. Thanks for the response!
On Sat, Mar 21, 2020 at 11:54 PM Liam Clarke
wrote:
> Hi Ryan,
>
> So long as you're not committing/storing offsets until that batch has been
> successfully p
Hi Ryan,
So long as you're not committing/storing offsets until that batch has been
successfully persisted you're fine (or if you're okay with some data loss
in the event of an app failing or being scaled down).
We use that approach in a couple of our apps that are persisting data into
datastores
You don't think it's weird if I just batch in memory manually do you? I
wrote a small snippet:
// Replicates Base Consumer
while (true) {
List records = consumer.poll();
// Batch records in-memory until criteria is satisfied
consumer.recordBuffer.addAll(records);
if (isNull(co
Hi Ryan,
If your end goal is just larger files on the server, you don't really need
to mess with the batching configs. You could just write multiple polls
worth of data to a single file.
On Fri, Mar 20, 2020 at 3:50 PM Liam Clarke
wrote:
> Hi Ryan,
>
> That'll be per poll.
>
> Kind regards,
>
Hi Ryan,
That'll be per poll.
Kind regards,
Liam Clarke
On Sat, 21 Mar. 2020, 11:41 am Ryan Schachte,
wrote:
> I do see the default for message.max.bytes is set to 1MB though. That would
> be for each record or each poll?
>
> On Fri, Mar 20, 2020 at 3:36 PM Ryan Schachte
> wrote:
>
> > Hi Li
I do see the default for message.max.bytes is set to 1MB though. That would
be for each record or each poll?
On Fri, Mar 20, 2020 at 3:36 PM Ryan Schachte
wrote:
> Hi Liam,
> We are running 2.3.1. I was hoping I wouldn't need to modify anything at
> the broker level since I do not have control/a
Hi Liam,
We are running 2.3.1. I was hoping I wouldn't need to modify anything at
the broker level since I do not have control/access to the broker config,
just the consumer configuration. Am I out of luck in that case?
On Fri, Mar 20, 2020 at 3:27 PM Liam Clarke
wrote:
> Hi Ryan,
>
> Firstly,
Hi Ryan,
Firstly, what version Kafka?
Secondly check the broker's message.max.bytes and the topic's
max.message.bytes, I suspect they're set a lot lower (or not at all) and
will override your fetch.min.bytes.
Cheers,
Liam Clarke
On Sat, 21 Mar. 2020, 11:09 am Ryan Schachte,
wrote:
> Hey guys
