Sorry I forgot to specify, this needs to go into your Connect worker configuration. On Fri, Jan 6, 2017 at 02:57 <david.frank...@bt.com> wrote:
> Hi Shikhar, > > I've just added this to ~config/consumer.properties in the Kafka folder > but it doesn't appear to have made any difference. Have I put it in the > wrong place? > > Thanks again, > David > > -----Original Message----- > From: Shikhar Bhushan [mailto:shik...@confluent.io] > Sent: 05 January 2017 18:12 > To: dev@kafka.apache.org > Subject: Re: KafkaConnect SinkTask::put > > Hi David, > > You can override the underlying consumer's `max.poll.records` setting for > this. E.g. > consumer.max.poll.records=500 > > Best, > > Shikhar > > On Thu, Jan 5, 2017 at 3:59 AM <david.frank...@bt.com> wrote: > > > Is there any way of limiting the number of events that are passed into > > the call to the put(Collection<SinkRecord>) method? > > > > I'm writing a set of events to Kafka via a source Connector/Task and > > reading these from a sink Connector/Task. > > If I generate of the order of 10k events the number of SinkRecords > > passed to the put method starts off very low but quickly rises in > > large increments such that 9k events are passed to a later invocation of > the put method. > > > > Furthermore, processing a large number of events in a single call (I'm > > writing to Elasticsearch) appears to cause the source task poll() > > method to timeout, raising a CommitFailedException which, > > incidentally, I can't see how to catch. > > > > Thanks for any help you can provide, > > David > > >
