[jira] [Created] (KAFKA-7442) forceUnmap mmap on linux

[JIRA]

翟玉勇 created KAFKA-7442:
--

 Summary: forceUnmap mmap on linux
 Key: KAFKA-7442
 URL: https://issues.apache.org/jira/browse/KAFKA-7442
 Project: Kafka
  Issue Type: Improvement
  Components: log
Affects Versions: 0.10.1.1
Reporter: 翟玉勇


when resize OffsetIndex or TimeIndex,We should force unmap mmap for linux 
platform

{code}
def resize(newSize: Int) {
inLock(lock) {
  val raf = new RandomAccessFile(_file, "rw")
  val roundedNewSize = roundDownToExactMultiple(newSize, entrySize)
  val position = mmap.position

  /* Windows won't let us modify the file length while the file is mmapped 
:-( */
  if(Os.isWindows)
forceUnmap(mmap)
  try {
raf.setLength(roundedNewSize)
mmap = raf.getChannel().map(FileChannel.MapMode.READ_WRITE, 0, 
roundedNewSize)
_maxEntries = mmap.limit / entrySize
mmap.position(position)
  } finally {
CoreUtils.swallow(raf.close())
  }
}
  }
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Accessing Topology Builder

[Jorge Esteban Quilcate Otoya]

Good to know, thanks Matthias!

You've mentioned a previous operator, but what about:
`peek().mapValues().peek()`, will both `peek`s be in the same thread as
well?

El mar., 25 sept. 2018 a las 23:14, Matthias J. Sax ()
escribió:

> Just for clarification:
>
> `peek()` would run on the same thread and the previous operator. Even
> if---strictly speaking---there is no public contract to guarantee this,
> it would be the case in the current implementation, and I also don't see
> any reason why this would change at any point in the future, because
> it's the most efficient implementation I can think of.
>
> -Matthias
>
> On 9/22/18 4:51 AM, Jorge Esteban Quilcate Otoya wrote:
> > Thanks, everyone!
> >
> > @Bill, the main issue with using `KStraem#peek()` is that AFAIK each
> `peek`
> > processor runs on a potentially different thread, then passing the trace
> > between them could be challenging. It will also require users to add
> these
> > operators themselves, which could be too cumbersome to use.
> >
> > @Guozhang and @John: I will first focus on creating the
> > `TracingProcessorSupplier` for instrumenting custom `Processors` and I
> will
> > keep the idea of a `ProcessorInterceptor` in the back of my head to see
> if
> > it make sense to propose a KIP for this.
> >
> > Thanks again for your feedback!
> >
> > Cheers,
> > Jorge.
> > El mié., 19 sept. 2018 a las 1:55, Bill Bejeck ()
> > escribió:
> >
> >> Jorge:
> >>
> >> I have a crazy idea off the top of my head.
> >>
> >> Would something as low-tech using KSteam.peek calls on either side of
> >> certain processors to record start and end times work?
> >>
> >> Thanks,
> >> Bill
> >>
> >> On Tue, Sep 18, 2018 at 4:38 PM Guozhang Wang 
> wrote:
> >>
> >>> Jorge:
> >>>
> >>> My suggestion was to let your users to implement on the
> >>> TracingProcessorSupplier
> >>> / TracingProcessor directly instead of the base-line ProcessorSupplier
> /
> >>> Processor. Would that work for you?
> >>>
> >>>
> >>> Guozhang
> >>>
> >>>
> >>> On Tue, Sep 18, 2018 at 8:02 AM, Jorge Esteban Quilcate Otoya <
> >>> quilcate.jo...@gmail.com> wrote:
> >>>
>  final StreamsBuilder builder = kafkaStreamsTracing.builder();Thanks
>  Guozhang and John.
> 
>  @Guozhang:
> 
> > I'd suggest to provide a
> > WrapperProcessorSupplier for the users than modifying
> > InternalStreamsTopology: more specifically, you can provide an
> > `abstract WrapperProcessorSupplier
> > implements ProcessorSupplier` and then let users to instantiate this
>  class
> > instead of the "bare-metal" interface. WDYT?
> 
>  Yes, in the gist, I have a class implementing `ProcessorSupplier`:
> 
>  ```
>  public class TracingProcessorSupplier implements
> >>> ProcessorSupplier  V> {
>    final KafkaTracing kafkaTracing;
>    final String name;
>    final ProcessorSupplier delegate;
> public TracingProcessorSupplier(KafkaTracing kafkaTracing,
>    String name, ProcessorSupplier delegate) {
>  this.kafkaTracing = kafkaTracing;
>  this.name = name;
>  this.delegate = delegate;
>    }
> @Override public Processor get() {
>  return new TracingProcessor<>(kafkaTracing, name, delegate.get());
>    }
>  }
>  ```
> 
>  My challenge is how to wrap Topology Processors created by
>  `StreamsBuilder#build` to make this instrumentation easy to adopt by
> >>> Kafka
>  Streams users.
> 
>  @John:
> 
> > The diff you posted only contains the library-side changes, and it's
> >>> not
> > obvious how you would use this to insert the desired tracing code.
> > Perhaps you could provide a snippet demonstrating how you want to use
>  this
> > change to enable tracing?
> 
>  My first approach was something like this:
> 
>  ```
>  final StreamsBuilder builder = kafkaStreamsTracing.builder();
>  ```
> 
>  Where `KafkaStreamsTracing#builder` looks like this:
> 
>  ```
>    public StreamsBuilder builder() {
>  return new StreamsBuilder(new Topology(new
>  TracingInternalTopologyBuilder(kafkaTracing)));
>    }
>  ```
> 
>  Then, once the builder creates a topology, `processors` will be
> wrapped
> >>> by
>  `TracingProcessorSupplier` described above.
> 
>  Probably this approach is too naive but works as an initial proof of
>  concept.
> 
> > Off the top of my head, here are some other approaches you might
>  evaluate:
> > * you mentioned interceptors. Perhaps we could create a
> > ProcessorInterceptor interface and add a config to set it.
> 
>  This sounds very interesting to me. Then we won't need to touch
> >> internal
>  API's, and just provide some configs. One challenge here is how to
> >> define
>  the hooks. In consumer/producer, lifecycle is clear,
> >>> `onConsumer`/`onSend`
>  and then `onCommit`/`onAck` methods

Re: [VOTE] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

[Satish Duggana]

Thanks for the KIP! +1 (non-binding)

On Tue, Sep 25, 2018 at 1:38 PM, Ismael Juma  wrote:
> Thanks, +1 (binding).
>
> Ismael
>
> On Thu, Sep 20, 2018 at 8:12 PM Chia-Ping Tsai  wrote:
>
>> KIP-336[1] has been merged so it is time to activate this thread
>> (KIP-331[2]). Last discussion is about "Should we add FunctionalInterface
>> annotation to Serializer and Deserializer". In discussion of KIP-336 we
>> mentioned that we probably add the default implementation for headless
>> method later. Hence, adding FunctionalInterface annotation is not suitable
>> now.
>>
>> KIP-331 has removed the change of adding FunctionalInterface annotation.
>> Please take a look again.
>>
>> [1]
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242
>> [2]
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
>>
>> Cheers,
>> Chia-Ping
>>
>>
>> On 2018/07/20 10:56:59, Ismael Juma  wrote:
>> > Part of the motivation for this KIP is to make these interfaces
>> functional
>> > interfaces. But I think that may not be desirable due to the method that
>> > passes headers. So, it doesn't make sense to discuss two separate changes
>> > to the same interfaces in isolation, we should figure out how we want
>> them
>> > to work holistically.
>> >
>> > Ismael
>> >
>> > On Fri, Jul 20, 2018 at 3:50 AM Chia-Ping Tsai 
>> wrote:
>> >
>> > > > The KIP needs 3 binding votes to pass.
>> > >
>> > > Thanks for the reminder. I will reopen the ballot box until we get 3
>> > > tickets.
>> > >
>> > > > I still think we should include the details of how things will look
>> like
>> > > > with the headers being passed to serializers/deserializers to ensure
>> > > > things actually make sense as a whole.
>> > >
>> > > This KIP is unrelated to the both methods - serialize() and
>> deserialize().
>> > > We won't add the default implementation to them in this kip. Please
>> correct
>> > > me if I didn't catch what you said.
>> > >
>> > > Cheers,
>> > > Chia-Ping
>> > >
>> > > On 2018/07/09 01:55:41, Ismael Juma  wrote:
>> > > > The KIP needs 3 binding votes to pass. I still think we should
>> include
>> > > the
>> > > > details of how things will look like with the headers being passed to
>> > > > serializers/deserializers to ensure things actually make sense as a
>> > > whole.
>> > > >
>> > > > Ismael
>> > > >
>> > > >
>> > > > On Sun, 8 Jul 2018, 18:31 Chia-Ping Tsai, 
>> wrote:
>> > > >
>> > > > > All,
>> > > > >
>> > > > > The 72 hours has passed. The vote result of KIP-313 is shown below.
>> > > > >
>> > > > > 1 binding vote (Matthias J. Sax)
>> > > > > 4 non-binding votes (John Roesler, Richard Yu, vito jeng and
>> Chia-Ping)
>> > > > >
>> > > > > Cheers,
>> > > > > Chia-Ping
>> > > > >
>> > > > > On 2018/07/05 14:45:01, Chia-Ping Tsai 
>> wrote:
>> > > > > > hi all,
>> > > > > >
>> > > > > > I would like to start voting on "KIP-331 Add default
>> implementation
>> > > to
>> > > > > close() and configure() for Serializer, Deserializer and Serde"
>> > > > > >
>> > > > > >
>> > > > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
>> > > > > >
>> > > > > > Cheers,
>> > > > > > Chia-Ping
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Re: [ANNOUNCE] New committer: Colin McCabe

[Jorge Esteban Quilcate Otoya]

Congratulations Colin!!

El mié., 26 sept. 2018 a las 5:54, Dongjin Lee ()
escribió:

> Congratulations!!
>
> Best,
> Dongjin
>
> On Wed, Sep 26, 2018 at 11:56 AM Satish Duggana 
> wrote:
>
> > Congratulations Colin!
> >
> >
> >
> > On Wed, Sep 26, 2018 at 5:52 AM, Vahid Hashemian
> >  wrote:
> > > Congratulations Colin!
> > >
> > > Regards.
> > > --Vahid
> > >
> > > On Tue, Sep 25, 2018 at 3:43 PM Colin McCabe 
> wrote:
> > >
> > >> Thanks, everyone!
> > >>
> > >> best,
> > >> Colin
> > >>
> > >>
> > >> On Tue, Sep 25, 2018, at 15:26, Robert Barrett wrote:
> > >> > Congratulations Colin!
> > >> >
> > >> > On Tue, Sep 25, 2018 at 1:51 PM Matthias J. Sax <
> > matth...@confluent.io>
> > >> > wrote:
> > >> >
> > >> > > Congrats Colin! The was over due for some time :)
> > >> > >
> > >> > > -Matthias
> > >> > >
> > >> > > On 9/25/18 1:51 AM, Edoardo Comar wrote:
> > >> > > > Congratulations Colin !
> > >> > > > --
> > >> > > >
> > >> > > > Edoardo Comar
> > >> > > >
> > >> > > > IBM Event Streams
> > >> > > > IBM UK Ltd, Hursley Park, SO21 2JN
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > From:   Ismael Juma 
> > >> > > > To: Kafka Users , dev <
> > >> dev@kafka.apache.org>
> > >> > > > Date:   25/09/2018 09:40
> > >> > > > Subject:[ANNOUNCE] New committer: Colin McCabe
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > Hi all,
> > >> > > >
> > >> > > > The PMC for Apache Kafka has invited Colin McCabe as a committer
> > and
> > >> we
> > >> > > > are
> > >> > > > pleased to announce that he has accepted!
> > >> > > >
> > >> > > > Colin has contributed 101 commits and 8 KIPs including
> significant
> > >> > > > improvements to replication, clients, code quality and testing.
> A
> > few
> > >> > > > highlights were KIP-97 (Improved Clients Compatibility Policy),
> > >> KIP-117
> > >> > > > (AdminClient), KIP-227 (Incremental FetchRequests to Increase
> > >> Partition
> > >> > > > Scalability), the introduction of findBugs and adding Trogdor
> > (fault
> > >> > > > injection and benchmarking tool).
> > >> > > >
> > >> > > > In addition, Colin has reviewed 38 pull requests and
> participated
> > in
> > >> more
> > >> > > > than 50 KIP discussions.
> > >> > > >
> > >> > > > Thank you for your contributions Colin! Looking forward to many
> > >> more. :)
> > >> > > >
> > >> > > > Ismael, for the Apache Kafka PMC
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > Unless stated otherwise above:
> > >> > > > IBM United Kingdom Limited - Registered in England and Wales
> with
> > >> number
> > >> > > > 741598.
> > >> > > > Registered office: PO Box 41, North Harbour, Portsmouth,
> Hampshire
> > >> PO6
> > >> > > 3AU
> > >> > > >
> > >> > >
> > >> > >
> > >>
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
> *github:  github.com/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> slideshare:
> www.slideshare.net/dongjinleekr
> *
>

[jira] [Created] (KAFKA-7443) OffsetOutOfRangeException in restoring state store from changelog topic when start offset of local checkpoint is smaller than that of changelog topic

[linyue li (JIRA)]

linyue li created KAFKA-7443:


 Summary: OffsetOutOfRangeException in restoring state store from 
changelog topic when start offset of local checkpoint is smaller than that of 
changelog topic
 Key: KAFKA-7443
 URL: https://issues.apache.org/jira/browse/KAFKA-7443
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.0.0, 2.1.0
Reporter: linyue li


When restoring local state store from a changelog topic in EOS, kafka stream 
will sometimes throw out the OffsetOutOfRangeException such as:
{code:java}
Restoring StreamTasks failed. Deleting StreamTasks stores to recreate from 
scratch.
org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of 
range with no configured reset policy for partitions: 
{AuditTrailBatch_PROD3-Dedup-key-store-changelog-32=75465112}
 at 
org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:950)
 at 
org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:470)
 at 
org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1249)
 at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1181)
 at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1157)
 at 
org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:89)
 at 
org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:321)
 at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:822)
 at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:765)
 at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:734){code}
 

This scenario occurs when changelog topic deleted the expired log segments 
according to the retention.ms, but the start offset in the local .checkpoint 
file is the position when the task last exits from this instance, which may be 
smaller than the updated beginning offset of changelog topic. Restoring store 
from start offset in checkpoint file will throw exception.

It can be reproduced as below (Kafka Stream runs in EOS):
 # task for topic partition test-1 is running on instance A. When task exits, 
kafka stream writes the last committed offset 100 for test-1 in checkpoint file.
 # task test-1 transfer to instance B.
 # During this time, the remote changelog topic for test-1 updates its start 
offset to 120 as the old log segment reaches retention time and is deleted.
 # After a while, task test-1 exits from instance B and resumes on instance A, 
and task restores local state store of A from checkpoint offset 100, which is 
smaller than the valid offset 120 of changelog topic. Such exception throws out.

When this exception occurs, kafka stream tries to reinitialize the task and 
intends to restore from beginning in catch block below. Unfortunately, this 
handle not work and the task keeps throwing  OffsetOutOfRangeException in the 
following restoring processes.
{code:java}
//org/apache/kafka/streams/processor/internals/StoreChangelogReader.java
//handle for OffsetOutOfRangeException in kafka stream

catch (final InvalidOffsetException recoverableException) {
 log.warn("Restoring StreamTasks failed. Deleting StreamTasks stores to 
recreate from scratch.", recoverableException);
 final Set partitions = recoverableException.partitions();
 for (final TopicPartition partition : partitions) {
   final StreamTask task = active.restoringTaskFor(partition);
   log.info("Reinitializing StreamTask {} for changelog {}", task, partition);

   needsInitializing.remove(partition);
   needsRestoring.remove(partition);

   task.reinitializeStateStoresForPartitions(recoverableException.partitions());
 }
 restoreConsumer.seekToBeginning(partitions);
}{code}
 

 Investigate why the handle for this exception not work, I found the root cause:

 Kafka stream registered state restorers in the variable stateRestorers, which 
is used to read /update the start and end offset for restoring local state 
store.
{code:java}
//org/apache/kafka/streams/processor/internals/StoreChangelogReader.java

private final Map stateRestorers = new 
HashMap<>();{code}
 

When the OffsetOutOfRangeException occurs, kafka stream should updates the 
checkpoint offset in the state restorer of this topic partition to 
"NO_CHECKPOINT" state, and the next time, task can restore from the beginning 
offset of remote changelog topic and resolve this issue.

But in catch block above, 
task.reinitializeStateStoresForPartitions(recoverableException.partitions()) 
not actually updates the checkpoint offset in stateRestorers, so the next time 
it still resumes from the original invalid offset and stuck in this exception.

I make some fix for this bug, by updating the checkpoint offset for this 
stateRestorer. and I v

Re: [DISCUSS] KIP-375: TopicCommand to use AdminClient

[Viktor Somogyi-Vass]

Hi Manikumar,

Thanks for raising this to my attention. I must have missed this in the
AdminClient, so you're right, we don't need this new protocol at all. I've
moved it to the rejected alternatives section.

Thanks,
Viktor

On Wed, Sep 26, 2018 at 8:24 AM Manikumar  wrote:

> Hi Viktor,
>
> We already have API (AdminClient.createPartitions,
> CreatePartitionsRequest/Response ) to increase the number of partitions of
> the topics.
> So, we may not need the protocol changes proposed in the KIP.  Let me know,
> If I am missing anything.
>
> Thanks,
>
>
> On Tue, Sep 25, 2018 at 2:45 PM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>
> wrote:
>
> > Hi Eno,
> >
> > Thanks for the question. Basically KAFKA-5561 aims for a bigger task: not
> > to add functionality to the existing TopicCommand but rewrite it in the
> > tools module in Java. KIP-375 only aims for extending the existing
> > TopicCommand's capability, so it is a smaller and backward compatible
> step
> > in the direction of having the tools communicating via the Kafka
> protocol.
> > Other commands, such as ConfigCommand (partly), ConsumerGroupCommand,
> > LogDirsCommand, DelegationTokenCommand also moved this direction, so I
> > think it makes sense to do it for TopicCommand as well.
> >
> > Cheers,
> > Viktor
> >
> > On Tue, Sep 25, 2018 at 10:48 AM Eno Thereska 
> > wrote:
> >
> > > This would be very useful.
> > > Could you clarify a bit the difference to
> > > https://issues.apache.org/jira/browse/KAFKA-5561 since I didn't get it
> > > from
> > > the JIRA notes. It's fine if you pick up that work but wanted to make
> > sure
> > > we're not duplicating efforts.
> > >
> > > Thanks
> > > Eno
> > >
> > > On Mon, Sep 24, 2018 at 8:26 PM, Viktor Somogyi-Vass <
> > > viktorsomo...@gmail.com> wrote:
> > >
> > > > Hi Gwen,
> > > >
> > > > Thanks for your feedback. It is the latter, so passing extra
> connection
> > > > properties for the admin client. I'll try to make that clearer in the
> > > KIP.
> > > > The same option name is used in the ConfigCommand, so that's why I
> > named
> > > it
> > > > "command-config".
> > > >
> > > > Cheers,
> > > > Viktor
> > > >
> > > >
> > > > On Mon, Sep 24, 2018 at 8:18 PM Gwen Shapira 
> > wrote:
> > > >
> > > > > The "use admin client" part is amazing and thank you.
> > > > >
> > > > > I'm confused about "commandConfig" - is this a list of
> configurations
> > > for
> > > > > use with --config option? Or a list of properties for connecting to
> > > > brokers
> > > > > (like SSL and such)? If the former, it seems unrelated.
> > > > >
> > > > > On Mon, Sep 24, 2018 at 7:25 AM Viktor Somogyi-Vass <
> > > > > viktorsomo...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I wrote up a relatively simple KIP about improving the Kafka
> > protocol
> > > > and
> > > > > > the TopicCommand tool to support the new Java based AdminClient
> and
> > > > > > hopefully to deprecate the Zookeeper side of it.
> > > > > >
> > > > > > I would be happy to receive some opinions about this. In general
> I
> > > > think
> > > > > > this would be an important addition as this is one of the few
> left
> > > but
> > > > > > important tools that still uses direct Zookeeper connection.
> > > > > >
> > > > > > Here is the link for the KIP:
> > > > > >
> > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 375%3A+TopicCommand+to+use+AdminClient
> > > > > >
> > > > > > Thanks,
> > > > > > Viktor
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Gwen Shapira*
> > > > > Product Manager | Confluent
> > > > > 650.450.2760 | @gwenshap
> > > > > Follow us: Twitter  | blog
> > > > > 
> > > > >
> > > >
> > >
> >
>

Kafka 7382 - Guarantee atleast one replica of partition be alive during create topic.

[Suman B N]

Team,

Issue KAFKA-7382  has
been fixed. Please review and merge. We need this fix in one of our
production clusters.

Pull request is here .

-- 
*Suman*
*OlaCabs*

Re: [ANNOUNCE] New committer: Colin McCabe

[Srinivas Reddy]

Congratulations Colin 👏

-
Srinivas

- Typed on tiny keys. pls ignore typos.{mobile app}

On Tue 25 Sep, 2018, 16:39 Ismael Juma,  wrote:

> Hi all,
>
> The PMC for Apache Kafka has invited Colin McCabe as a committer and we are
> pleased to announce that he has accepted!
>
> Colin has contributed 101 commits and 8 KIPs including significant
> improvements to replication, clients, code quality and testing. A few
> highlights were KIP-97 (Improved Clients Compatibility Policy), KIP-117
> (AdminClient), KIP-227 (Incremental FetchRequests to Increase Partition
> Scalability), the introduction of findBugs and adding Trogdor (fault
> injection and benchmarking tool).
>
> In addition, Colin has reviewed 38 pull requests and participated in more
> than 50 KIP discussions.
>
> Thank you for your contributions Colin! Looking forward to many more. :)
>
> Ismael, for the Apache Kafka PMC
>

Jenkins build is back to normal : kafka-trunk-jdk10 #522

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk8 #2992

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAFKA-7428: ConnectionStressSpec: add "action", allow multiple 
clients

[rajinisivaram] MINOR: Fix broken link in security.html (#5670)

--
[...truncated 2.72 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopic[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopic[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled 

[DISCUSS] KIP-375 Kafka Clients - make Metadata#TOPIC_EXPIRY_MS configurable

[Pavel Moukhataev]

https://cwiki.apache.org/confluence/display/KAFKA/KIP-375%3A+Kafka+Clients+-+make+Metadata%23TOPIC_EXPIRY_MS+configurable

I'd like to introduce new feature for kafka client:
Making org.apache.kafka.clients.Metadata#TOPIC_EXPIRY_MS configurable

The problem is: if application sends records to some topic rarely then
topic metadata gets expired and sending thread is blocked by waiting topic
metadata.

Easy fix of this exact problem is to make Metadata#TOPIC_EXPIRY_MS
configurable and set it value higher than record sending period.

-- 
Pavel
+7-903-258-5544
skype://pavel.moukhataev

Re: [DISCUSS] KIP-375 Kafka Clients - make Metadata#TOPIC_EXPIRY_MS configurable

[Viktor Somogyi-Vass]

Hi Pavel,

May I kindly ask you to increment your KIP number? :)
As it turns out I have already created one with this number a little bit
earlier. Apologies for the inconvenience.

Viktor

On Wed, Sep 26, 2018 at 4:48 PM Pavel Moukhataev 
wrote:

>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-375%3A+Kafka+Clients+-+make+Metadata%23TOPIC_EXPIRY_MS+configurable
>
> I'd like to introduce new feature for kafka client:
> Making org.apache.kafka.clients.Metadata#TOPIC_EXPIRY_MS configurable
>
> The problem is: if application sends records to some topic rarely then
> topic metadata gets expired and sending thread is blocked by waiting topic
> metadata.
>
> Easy fix of this exact problem is to make Metadata#TOPIC_EXPIRY_MS
> configurable and set it value higher than record sending period.
>
> --
> Pavel
> +7-903-258-5544
> skype://pavel.moukhataev
>

Re: [ANNOUNCE] New committer: Colin McCabe

[Konstantine Karantasis]

Well deserved! Congratulations Colin.

-Konstantine

On Wed, Sep 26, 2018 at 4:57 AM Srinivas Reddy 
wrote:

> Congratulations Colin 👏
>
> -
> Srinivas
>
> - Typed on tiny keys. pls ignore typos.{mobile app}
>
> On Tue 25 Sep, 2018, 16:39 Ismael Juma,  wrote:
>
> > Hi all,
> >
> > The PMC for Apache Kafka has invited Colin McCabe as a committer and we
> are
> > pleased to announce that he has accepted!
> >
> > Colin has contributed 101 commits and 8 KIPs including significant
> > improvements to replication, clients, code quality and testing. A few
> > highlights were KIP-97 (Improved Clients Compatibility Policy), KIP-117
> > (AdminClient), KIP-227 (Incremental FetchRequests to Increase Partition
> > Scalability), the introduction of findBugs and adding Trogdor (fault
> > injection and benchmarking tool).
> >
> > In addition, Colin has reviewed 38 pull requests and participated in more
> > than 50 KIP discussions.
> >
> > Thank you for your contributions Colin! Looking forward to many more. :)
> >
> > Ismael, for the Apache Kafka PMC
> >
>

Re: [ANNOUNCE] New committer: Colin McCabe

[Yishun Guan]

Congrats! -Yishun

On Wed, Sep 26, 2018, 10:04 AM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Well deserved! Congratulations Colin.
>
> -Konstantine
>
> On Wed, Sep 26, 2018 at 4:57 AM Srinivas Reddy  >
> wrote:
>
> > Congratulations Colin 👏
> >
> > -
> > Srinivas
> >
> > - Typed on tiny keys. pls ignore typos.{mobile app}
> >
> > On Tue 25 Sep, 2018, 16:39 Ismael Juma,  wrote:
> >
> > > Hi all,
> > >
> > > The PMC for Apache Kafka has invited Colin McCabe as a committer and we
> > are
> > > pleased to announce that he has accepted!
> > >
> > > Colin has contributed 101 commits and 8 KIPs including significant
> > > improvements to replication, clients, code quality and testing. A few
> > > highlights were KIP-97 (Improved Clients Compatibility Policy), KIP-117
> > > (AdminClient), KIP-227 (Incremental FetchRequests to Increase Partition
> > > Scalability), the introduction of findBugs and adding Trogdor (fault
> > > injection and benchmarking tool).
> > >
> > > In addition, Colin has reviewed 38 pull requests and participated in
> more
> > > than 50 KIP discussions.
> > >
> > > Thank you for your contributions Colin! Looking forward to many more.
> :)
> > >
> > > Ismael, for the Apache Kafka PMC
> > >
> >
>

Re: [EXTERNAL] [DISCUSS] KIP-310: Add a Kafka Source Connector to Kafka Connect

[Konstantine Karantasis]

Hi Rhys,

thanks for the proposal and apologies for the late feedback. Utilizing
Connect to mirror Kafka topics is definitely a plausible proposal for a
very useful use case.

However, I don't think the apache/kafka repository is the right place to
host such a Connector. Currently, no full-featured, production-ready
connectors are hosted in AK. The only two connectors shipped with AK
(FileStreamSourceConnector and FileStreamSinkConnector) are there to
demonstrate implementations only as examples.

I find this approach very appealing. AK focuses on providing the core
infrastructure for Connect, that is required in every Kafka Connect
deployment, as well as offering the means to generically install, deploy
and operate connectors. But all the connectors reside outside AK and
comprise a vibrant ecosystem of open source and proprietary components
that, essentially - even for the most useful and ubiquitous of the
connectors - are optional for users to install and use. This seems simple
and flexible, both in terms of releasing and using/deploying software
related to Kafka Connect. I might even say that I'd be in favor of
extending this approach to all the Connect components, including
Transformations and Converters.

I'm aware that MirrorMaker is part of AK, but to me this refers to the
early days of Apache Kafka, when the size of the project and the ecosystem
was smaller, Connect and Streams had not been implemented yet, and
mirroring topics between Kafka clusters was already a basic need. With a
much more rich ecosystem now and more sizable and well defined packages in
AK, I think the approach that decouples connectors from the Connect
framework itself is a good one.

In my opinion, the fact that this connector targets Kafka itself as a
source is not an adequate reason to include it in apache/kafka within the
Connect framework. It seems it can evolve naturally, as every other
connector, in its own repository.

Regards,
Konstantine


On Sat, Aug 4, 2018 at 7:20 PM McCaig, Rhys  wrote:

> Hi All,
>
> If there are no further comments on this KIP I’ll start a vote early this
> week.
>
> Rhys
>
> On Aug 1, 2018, at 12:32 AM, McCaig, Rhys  > wrote:
>
> Hi All,
>
> I’ve updated the proposal to include the improvements suggested by
> Stephane.
>
> I have also submitted a PR to implement this functionality into Kafka.
> https://github.com/apache/kafka/pull/5438
>
> I don’t have a benchmark against MirrorMaker yet, as I only currently have
> a local docker stack available to me, though I have seen very good
> performance in that test stack (200k messages/sec@100bytes on limited
> compute resource containers). Further benchmarking might take a few days.
>
> Review and comments would be appreciated.
>
> Cheers,
> Rhys
>
>
> On Jun 18, 2018, at 9:00 AM, McCaig, Rhys  > wrote:
>
> Hi Stephane,
>
> Thanks for your feedback and apologies for the delay in my response.
>
> Are there any performance benchmarks against Mirror Maker available? I'm
> interested to know if this is more performant / scalable.
> Regarding the implementation, here's some feedback:
>
>
> Currently I don’t have any performance benchmarks, but I think this is a
> great idea, ill see if I can set up something one the next week or so.
>
> - I think it's worth mentioning that this solution does not rely on
> consumer groups, and therefore tracking progress may be tricky. Can you
> think of a way to expose that?
>
> This is a reasonable concern. I’m not sure how to track this other than
> looking at the Kafka connect offsets. Once a messages is passed to the
> framework, I'm unaware of a way to get at the commit offsets on the
> producer side. Any thoughts?
>
> - Some code can be in config Validator I believe:
>
> https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47
>
> - I think your kip mentions `source.admin.` and `source.consumer.` but I
> don't see it reflected yet in the code
>
> - Is there a way to be flexible and merge list and regex, or offer the two
> simultaneously ? source_topics=my_static_topic,prefix.* ?
>
> Agree on all of the above - I will incorporate into the code later this
> week as ill get some time back to work on this.
>
> Cheers,
> Rhys
>
>
>
> On Jun 6, 2018, at 7:16 PM, Stephane Maarek <
> steph...@simplemachines.com.au>
> wrote:
>
> Hi Rhys,
>
> I think this will be a great addition.
>
> Are there any performance benchmarks against Mirror Maker available? I'm
> interested to know if this is more performant / scalable.
> Regarding the implementation, here's some feedback:
>
> - I think it's worth mentioning that this solution does not rely on
> consumer groups, and therefore tracking progress may be tricky. Can you
> think of a way to expose that?
>
>
> - Some code can be in config Validator I believe:
>
> https://github.com/Comcast

Re: [VOTE] KIP-310: Add a Kafka Source Connector to Kafka Connect

[Konstantine Karantasis]

Hi Rhys,

I just replied in the discussion thread for KIP-310 with my concerns.
Thanks for submitting this proposal.

-Konstantine


On Tue, Sep 11, 2018 at 12:53 AM McCaig, Rhys 
wrote:

> Hi All,
>
> Bumping this again.
> Can I get feedback from some binding vote holders on this KIP? I think its
> fairly straightforward KIP and worthwhile addition to Kafka Connect.
>
> Cheers,
> Rhys
>
>
>
> > On Sep 4, 2018, at 12:32 PM, McCaig, Rhys 
> wrote:
> >
> > Bumping this thread.
> >
> >> On Aug 14, 2018, at 11:00 AM, McCaig, Rhys 
> wrote:
> >>
> >> Bumping this thread. Looking for some binding votes or further request
> for discussion.
> >>
> >>> On Aug 10, 2018, at 12:38 PM, McCaig, Rhys 
> wrote:
> >>>
> >>> Thanks Stephane!
> >>>
> >>> If there is a desire for further discussion I am certainly open to
> reverting this to a discussion thread. For now I’ll keep this vote open
> until we get either 3 binding votes or further request for discussion from
> the community.
> >>>
> >>> Do you have any additional thoughts on the KIP you’d like to add?
> >>>
> >>> Cheers,
> >>> Rhys
> >>>
>  On Aug 10, 2018, at 2:14 AM, Stephane Maarek <
> steph...@simplemachines.com.au> wrote:
> 
>  Hi Rhys,
> 
>  Overall I'm +1 (non binding), but you're going to need 3 binding
> votes for
>  this KIP to pass.
>  I don't feel there has been enough discussion on this from the
> community.
>  Can we get some input from other people?
> 
>  Thanks for starting the vote nonetheless :)
>  Stephane
> 
>  On 8 August 2018 at 20:28, McCaig, Rhys 
> wrote:
> 
> > Hi
> >
> > Could we get a couple of votes on this KIP - voting closes in 24
> hours.
> >
> > Thanks,
> >
> > Rhys
> >
> >> On Aug 6, 2018, at 11:51 AM, McCaig, Rhys 
> > wrote:
> >>
> >> Hi All,
> >>
> >> I’m starting a vote on KIP-310: Add a Kafka Source Connector to
> Kafka
> > Connect
> >>
> >> KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 310%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect > ps://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 310:+Add+a+Kafka+Source+Connector+to+Kafka+Connect>
> >> Discussion Thread: http://mail-archives.apache.
> > org/mod_mbox/kafka-dev/201808.mbox/%3c17E8D696-E51C-4BEB-
> > bd70-9324d4b53...@comcast.com%3e > apache.org/mod_mbox/kafka-dev/201808.mbox/<17E8D696-E51C-
> > 4beb-bd70-9324d4b53...@comcast.com>>
> >>
> >> Cheers,
> >> Rhys
> >
> >
> >>>
> >>
> >
>
>

Re: [DISCUSS] Make org.apache.kafka.clients.Metadata#TOPIC_EXPIRY_MS configurable

[Dongjin Lee]

Hi Pavel,

Thanks for a great proposal. However, the code freeze of 2.1.0 is imminent,
so it seems like all the committers are on working frenzy. How about
discussing this KIP after the release of 2.1.0? I promise, I will reboot
this discussion as soon as 2.1.0 is released. I also experienced a similar
situation in the past.

Best,
Dongjin

On Mon, Sep 24, 2018 at 7:48 PM Pavel Moukhataev 
wrote:

> I'd like to introduce new feature for kafka client:
> Making org.apache.kafka.clients.Metadata#TOPIC_EXPIRY_MS configurable
> Here is KPI
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-375%3A+Make+org.apache.kafka.clients.Metadata%23TOPIC_EXPIRY_MS+configurable
>
> The problem is: if application sends records to some topic rarely then
> topic metadata gets expired and sending thread is blocked to wait topic
> metadata.
>
> Easy fix is to make TOPIC_EXPIRY_MS configurable.
>
> --
> Pavel
> +7-903-258-5544
> skype://pavel.moukhataev
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*

*github:  github.com/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
slideshare:
www.slideshare.net/dongjinleekr
*

[jira] [Created] (KAFKA-7444) expose connector, task IDs to SinkTasks

[Ryanne Dolan (JIRA)]

Ryanne Dolan created KAFKA-7444:
---

 Summary: expose connector, task IDs to SinkTasks
 Key: KAFKA-7444
 URL: https://issues.apache.org/jira/browse/KAFKA-7444
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 2.0.0
Reporter: Ryanne Dolan


SinkTasks don't have access to their ConnectorTaskId, which would be useful for 
logging and for using SinkUtils.consumerGroupId(id).

 

I suggest an id() method be added to SinkTaskContext and WorkerSinkTaskContext.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7437) Store leader epoch in offset commit metadata

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-7437.
---
   Resolution: Fixed
 Reviewer: Rajini Sivaram
Fix Version/s: 2.1.0

> Store leader epoch in offset commit metadata
> 
>
> Key: KAFKA-7437
> URL: https://issues.apache.org/jira/browse/KAFKA-7437
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.1.0
>
>
> This patch implements the changes described in KIP-320 for the persistence of 
> leader epoch information in the offset commit metadata: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-320%3A+Allow+fetchers+to+detect+and+handle+log+truncation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk8 #2993

[Apache Jenkins Server]

See 

Re: [EXTERNAL] [DISCUSS] KIP-310: Add a Kafka Source Connector to Kafka Connect

[McCaig, Rhys]

Hi Konstantine,

Thank you for your thoughtful comments!

> However, I don't think the apache/kafka repository is the right place to
> host such a Connector. 


> I find this approach very appealing. AK focuses on providing the core
> infrastructure for Connect, that is required in every Kafka Connect
> deployment, as well as offering the means to generically install, deploy
> and operate connectors.

I personally flip-flopped on this with similar thoughts with this when I 
initially considered raising a KIP for this functionality. 

When I initially developed a Kafka source connector, this was out of necessity 
- MirrorMaker requires zkconnect strings, which I didn't have access to for the 
source cluster, and Confluent’s proprietary connector also requried zk 
connections - though it has now been updated to remove this limitation. 

While I understand the point of view that MirrorMaker refers to the early days 
of Apache Kafka, it has become a critical tool for replicating data across 
Kafka clusters in for a large portion of the community who are managing Kafka 
at scale. As such, I suspect that there is a lot of interest in the Kafka 
project supporting topic replication across clusters. While one approach (which 
I don’t have the knowledge or time to address) could be to include it as a core 
component of Kafka itself (such as Apache Pulsar’s global topics), my view is 
that at this point in time, Kafka Connect is considered *the* way to ship data 
in and our of a specific Kafka cluster, regardless of the external system. 

I’d welcome further discussion on whether the community thinks is the right 
approach for the Kafka project to take, in regards to handling Kafka topic 
mirroring. I *think* that its important and common enough, that there should be 
support in the project - and MirrorMaker is, as you mention, showing its age. 

Cheers,
Rhys




> On Sep 26, 2018, at 10:42 AM, Konstantine Karantasis 
>  wrote:
> 
> Hi Rhys,
> 
> thanks for the proposal and apologies for the late feedback. Utilizing
> Connect to mirror Kafka topics is definitely a plausible proposal for a
> very useful use case.
> 
> However, I don't think the apache/kafka repository is the right place to
> host such a Connector. Currently, no full-featured, production-ready
> connectors are hosted in AK. The only two connectors shipped with AK
> (FileStreamSourceConnector and FileStreamSinkConnector) are there to
> demonstrate implementations only as examples.
> 
> I find this approach very appealing. AK focuses on providing the core
> infrastructure for Connect, that is required in every Kafka Connect
> deployment, as well as offering the means to generically install, deploy
> and operate connectors. But all the connectors reside outside AK and
> comprise a vibrant ecosystem of open source and proprietary components
> that, essentially - even for the most useful and ubiquitous of the
> connectors - are optional for users to install and use. This seems simple
> and flexible, both in terms of releasing and using/deploying software
> related to Kafka Connect. I might even say that I'd be in favor of
> extending this approach to all the Connect components, including
> Transformations and Converters.
> 
> I'm aware that MirrorMaker is part of AK, but to me this refers to the
> early days of Apache Kafka, when the size of the project and the ecosystem
> was smaller, Connect and Streams had not been implemented yet, and
> mirroring topics between Kafka clusters was already a basic need. With a
> much more rich ecosystem now and more sizable and well defined packages in
> AK, I think the approach that decouples connectors from the Connect
> framework itself is a good one.
> 
> In my opinion, the fact that this connector targets Kafka itself as a
> source is not an adequate reason to include it in apache/kafka within the
> Connect framework. It seems it can evolve naturally, as every other
> connector, in its own repository.
> 
> Regards,
> Konstantine
> 
> 
> On Sat, Aug 4, 2018 at 7:20 PM McCaig, Rhys  wrote:
> 
>> Hi All,
>> 
>> If there are no further comments on this KIP I’ll start a vote early this
>> week.
>> 
>> Rhys
>> 
>> On Aug 1, 2018, at 12:32 AM, McCaig, Rhys > > wrote:
>> 
>> Hi All,
>> 
>> I’ve updated the proposal to include the improvements suggested by
>> Stephane.
>> 
>> I have also submitted a PR to implement this functionality into Kafka.
>> https://github.com/apache/kafka/pull/5438
>> 
>> I don’t have a benchmark against MirrorMaker yet, as I only currently have
>> a local docker stack available to me, though I have seen very good
>> performance in that test stack (200k messages/sec@100bytes on limited
>> compute resource containers). Further benchmarking might take a few days.
>> 
>> Review and comments would be appreciated.
>> 
>> Cheers,
>> Rhys
>> 
>> 
>> On Jun 18, 2018, at 9:00 AM, McCaig, Rhys > > wrote:
>> 
>> Hi Stephane,
>> 
>> Than

[DISCUSSION] KIP-376: Implement AutoClosable on appropriate classes that has close()

[Yishun Guan]

Hi All,

Here is a trivial KIP:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=93325308

Suggestions are welcome.

Thanks,
Yishun

Re: [DISCUSSION] KIP-376: Implement AutoClosable on appropriate classes that has close()

[Bill Bejeck]

Hi Yishun,

Thanks for the KIP, seems like a useful addition.

Just a couple of minor comments.

Can you list all the changes in the list under "Public Interfaces" in the
"Proposed Changes" section that way it's clear what's changing?   I realize
they all will be very similar, but it's better to be explicit with the
proposed changes in a KIP

Also, you are proposing changes across several components and I'm not sure
if that is possible in a single KIP, but I could very well be wrong on this
one, so we'll see what others say.

Thanks,
Bill

On Wed, Sep 26, 2018 at 7:44 PM Yishun Guan  wrote:

> Hi All,
>
> Here is a trivial KIP:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=93325308
>
> Suggestions are welcome.
>
> Thanks,
> Yishun
>

Re: [DISCUSSION] KIP-376: Implement AutoClosable on appropriate classes that has close()

[Yishun Guan]

Hi Bill,

I see, i just updated the KIP with all the changes. And your second
concern make sense - if that's what everyone think, we can start with
changing KafkaStream first.

Thanks,
Yishun
On Wed, Sep 26, 2018 at 5:18 PM Bill Bejeck  wrote:
>
> Hi Yishun,
>
> Thanks for the KIP, seems like a useful addition.
>
> Just a couple of minor comments.
>
> Can you list all the changes in the list under "Public Interfaces" in the
> "Proposed Changes" section that way it's clear what's changing?   I realize
> they all will be very similar, but it's better to be explicit with the
> proposed changes in a KIP
>
> Also, you are proposing changes across several components and I'm not sure
> if that is possible in a single KIP, but I could very well be wrong on this
> one, so we'll see what others say.
>
> Thanks,
> Bill
>
> On Wed, Sep 26, 2018 at 7:44 PM Yishun Guan  wrote:
>
> > Hi All,
> >
> > Here is a trivial KIP:
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=93325308
> >
> > Suggestions are welcome.
> >
> > Thanks,
> > Yishun
> >

Re: [DISCUSSION] KIP-376: Implement AutoClosable on appropriate classes that has close()

[Chia-Ping Tsai]

hi Yishun

Thanks for nice KIP!

Q1)
Why VerifiableProducer extend Closeable rather than AutoCloseable?

Q2)
I grep project and then noticed there are other close methods but do not 
implement AutoCloseable.
For example:
1) WorkerConnector
2) MemoryRecordsBuilder
3) MetricsReporter
4) ExpiringCredentialRefreshingLogin
5) KafkaChannel
6) ConsumerInterceptor
7) SelectorMetrics
8) HeartbeatThread

Cheers,
Chia-Ping


On 2018/09/26 23:44:31, Yishun Guan  wrote: 
> Hi All,
> 
> Here is a trivial KIP:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=93325308
> 
> Suggestions are welcome.
> 
> Thanks,
> Yishun
> 

Re: [DISCUSS] KIP-374: Add '--help' option to all available Kafka CLI commands

[Srinivas Reddy]

Any thoughts about this KIP ?

-
Srinivas

- Typed on tiny keys. pls ignore typos.{mobile app}

On Fri 21 Sep, 2018, 16:07 Srinivas Reddy, 
wrote:

> Hi All,
>
> I'd like to start a discussion on KIP-374 which is to add "--help" option
> for all the possible CLI commands in Kafka. right now, few commands support
> it and few does not. We brings up this KIP to provide consistent user
> experience.
>
> More details added in this KIP page(including code changes):
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-374%3A+Add+%27--help%27+option+to+all+available+Kafka+CLI+commands
>
> I look forward to feedback and comments.
>
>
>
> --
> Srinivas Reddy
>
> http://mrsrinivas.com/
>
>
> (Sent via gmail web)
>

Re: [DISCUSS] KIP-374: Add '--help' option to all available Kafka CLI commands

[Dongjin Lee]

Hi Srinivas,

Thanks for your proposal. It sounds reasonable to me. However, the code
freeze of 2.1.0 is imminent so it seems like all the committers are on
working frenzy on it. How about to postpone reboot this discussion after
the release of 2.1.0? I will kick off the thread as soon as the release is
completed.

Thanks,
Dongjin

On Thu, Sep 27, 2018 at 1:31 PM Srinivas Reddy 
wrote:

> Any thoughts about this KIP ?
>
> -
> Srinivas
>
> - Typed on tiny keys. pls ignore typos.{mobile app}
>
> On Fri 21 Sep, 2018, 16:07 Srinivas Reddy, 
> wrote:
>
> > Hi All,
> >
> > I'd like to start a discussion on KIP-374 which is to add "--help" option
> > for all the possible CLI commands in Kafka. right now, few commands
> support
> > it and few does not. We brings up this KIP to provide consistent user
> > experience.
> >
> > More details added in this KIP page(including code changes):
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-374%3A+Add+%27--help%27+option+to+all+available+Kafka+CLI+commands
> >
> > I look forward to feedback and comments.
> >
> >
> >
> > --
> > Srinivas Reddy
> >
> > http://mrsrinivas.com/
> >
> >
> > (Sent via gmail web)
> >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*

*github:  github.com/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
slideshare:
www.slideshare.net/dongjinleekr
*