t;>>> +1 (non-binding). I haven’t contributed to the discussion but I’ve
> been
> >>> following - it’ll definitely make my team’s life easier.
> >>>>
> >>>>> On Sep 20, 2019, at 11:36 AM, Jukka Karvanen <
> >>>
(non-binding). I haven’t contributed to the discussion but I’ve been
>>> following - it’ll definitely make my team’s life easier.
>>>>
>>>>> On Sep 20, 2019, at 11:36 AM, Jukka Karvanen <
>>> jukka.karva...@jukinimi.com> wrote:
>>>>&
gt; > jukka.karva...@jukinimi.com> wrote:
> > >>
> > >> Hi all,
> > >>
> > >> I would like to start vote on KIP-470:
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
> > >>
> > >>
> > >> Regards,
> > >>
> > >> Jukka
> >
> >
>
> --
> -- Guozhang
>
ing - it’ll definitely make my team’s life easier.
> >
> >> On Sep 20, 2019, at 11:36 AM, Jukka Karvanen <
> jukka.karva...@jukinimi.com> wrote:
> >>
> >> Hi all,
> >>
> >> I would like to start vote on KIP-470:
> >>
> https://c
;> wrote:
>>
>> Hi all,
>>
>> I would like to start vote on KIP-470:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
>>
>>
>> Regards,
>>
>> Jukka
signature.asc
Description: OpenPGP digital signature
fluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
>
>
> Regards,
>
> Jukka
Hi all,
I would like to start vote on KIP-470:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
Regards,
Jukka
gt; guess my
> > >>>>>>>>> own preference is 5.
> > >>>>>>>>>
> > >>>>>>>>> It seems like the migration pain is a one-time concern vs.
> having
> > >> more
> > >>>>>&
;>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> -John
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
>>>>>>>>>> Hi Matthias,
>>>>>>>>>>
>>>>>>>>>> Generally I think using Instant and Duration make the test more
>>>>>>>> readable
>>>>>>>>>> and that's why I modified KIP based on your suggestion.
>>>>>&
>>>>>>>> complicated.
> >>>>>>>>
> >>>>>>>> What I tried to say about the migration is the lines without
> >>>>> timestamp
> >>>>>> or
> >>>>>>>> i
long timestamp is supported can be migrated mainly with search &
>>>>>>>> recplace:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>> testDriver.pipeInput(recordFactory.create(Wor
t;>
>>>>>>>
>>>>>>
>>>>>
>>>> testDriver.pipeInput(recordFactory.create(WordCountLambdaExample.inputTopic,
>>>>>>> nullKey, "Hello", 1L));
>>>>>>>
>>>>>>> -
CountLambdaExample.inputTopic,
> >> > > > nullKey, "Hello", 1L));
> >> > > >
> >> > > > ->
> >> > > >
> >> > > > *inputTopic1*.pipeInput(nullKey, "Hello", Instant.ofEpochMilli(1L));
>
ot;, Instant.ofEpochMilli(1L));
>> > > >
>> > > > Also if you need to convert arbitrary long timestamps to proper time
>> > > > Instants, it require you need to understand the logic of the test.
>> So
>> > > > mechanical search & repla
gt; > Instants, it require you need to understand the logic of the test. So
> > > > mechanical search & replace is not possible.
> > > >
> > > >
> > > > I see there are several alternatives for long vs Instant / Duration:
> > > >
pages/viewpage.action?pageId=119550011
> > >
> > > (startTimestampMs, autoAdvanceMs as parameter of createInputTopic
> > > instead of configureTiming)
> > >
> > > 2. Auto timestamping configured with Instant and Duration, pipeInput
> > > and Te
; > 2. Auto timestamping configured with Instant and Duration, pipeInput
> > and TestRecord with long:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120722523
> >
> >
> > 3. (CURRENT) Auto timestamping configured with Instant and
InputTopic
> instead of configureTiming)
>
> 2. Auto timestamping configured with Instant and Duration, pipeInput
> and TestRecord with long:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120722523
>
>
> 3. (CURRENT) Auto timestamping configured with Instant a
ENT) Auto timestamping configured with Instant and Duration,
pipeInput and TestRecord with Instant, version with long deprecated:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
4. Auto timestamping configured with Inst
ate separate object for each input and output
>>>> topic
>>>>>>> you are using. The topic name is given to createInput/OutputTopic
>>> when
>>>>>>> initialize topic object.
>>>>>>>
>>
putTopic(OUTPUT_TOPIC_MAP, stringSerde,
> > > >>> longSerde);
> > > >>> inputTopic1.pipeInput(1L, "Hello");
> > > >>> assertThat(outputTopic1.readKeyValue(), equalTo(new KeyValue<>(1L,
> > > >>> "Hello")))
lTo(new
> KeyValue<>("Hello",
> > >>> 1L)));
> > >>> inputTopic2.pipeInput(1L, "Hello");
> > >>>
> > >>>
> > >>> Jukka
> > >>>
> > >>> to 20. kesäk. 2
t; Thanks for writing the KIP, I have a couple of quick questions:
> >>>>
> >>>> 1) Is "TestRecord" an existing class that you propose to piggy-back
> on?
> >>>> Right now we have a scala TestRecord case class but I doubt that was
> >> your
> >>>&g
ipeInput how to
> > > determine
> > > > which topic this record should be pipe to?
> > > >
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Jun 17, 2019 at 1:34 PM John Roesler
> > wrote:
> > > >
&
th
> > > `createInput/OutputTopic`? If not, when we call pipeInput how to
> > determine
> > > which topic this record should be pipe to?
> > >
> > >
> > > Guozhang
> > >
> > > On Mon, Jun 17, 2019 at 1:34
;> your
>>>> proposal, or are you proposing to add a new Java class?
>>>>
>>>> 2) Would the new API only allow a single input / output topic with
>>>> `createInput/OutputTopic`? If not, when we call pipeInput how to
>>> determine
>&g
which topic this record should be pipe to?
> > >
> > >
> > > Guozhang
> > >
> > > On Mon, Jun 17, 2019 at 1:34 PM John Roesler
> wrote:
> > >
> > > > Woah, I wasn't aware of that Hamcrest test style. Awesome!
> > > >
> > > >
> > On Mon, Jun 17, 2019 at 1:34 PM John Roesler wrote:
> >
> > > Woah, I wasn't aware of that Hamcrest test style. Awesome!
> > >
> > > Thanks for the updates. I look forward to hearing what others think.
> > >
> > > -John
> > &
> >
> > Thanks for the updates. I look forward to hearing what others think.
> >
> > -John
> >
> > On Mon, Jun 17, 2019 at 4:12 AM Jukka Karvanen
> > wrote:
> > >
> > > Wiki page updated:
> > >
> >
> https:
est style. Awesome!
>
> Thanks for the updates. I look forward to hearing what others think.
>
> -John
>
> On Mon, Jun 17, 2019 at 4:12 AM Jukka Karvanen
> wrote:
> >
> > Wiki page updated:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+Top
logyTestDriver+test+input+and+output+usability+improvements
>
>
> ClientRecord removed and replaced with TestRecord in method calls.
> TestRecordFactory removed (time tracking functionality to be included to
> TestInputTopic)
> OutputVerifier deprecated
> TestRecord topic remov
Wiki page updated:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
ClientRecord removed and replaced with TestRecord in method calls.
TestRecordFactory removed (time tracking functionality to be included to
> >
> > > > The think I like most about it is that it mirrors the mental model
> > > > that people already have from Kafka Streams, in which you write to an
> > > > "input topic" and then get your results from an "output topic". As a
> >
utput topic". As a
> > > side benefit, the input and output topics in the proposal also close
> > > over the serdes, which makes it much less boilerplate for test code.
> > >
> > > If I can offer one suggestion for change, I'm not sure I'm
s like this
> > unnecessarily complicates the main (non-testing) data model. It seems
> > like it would be sufficient, and just as ergonomic, to have the input
> > topic accept ProducerRecords and the output topic return
> > ConsumerRecords. I'm open to discussion on this point, t
topic accept ProducerRecords and the output topic return
> ConsumerRecords. I'm open to discussion on this point, though.
>
> Thanks for the proposal, Jukka!
> -John
>
> On Mon, May 20, 2019 at 7:59 AM Jukka Karvanen
> wrote:
> >
> > Hi All,
> >
> &
accept ProducerRecords and the output topic return
ConsumerRecords. I'm open to discussion on this point, though.
Thanks for the proposal, Jukka!
-John
On Mon, May 20, 2019 at 7:59 AM Jukka Karvanen
wrote:
>
> Hi All,
>
> I would like to start the discussion on KIP-470: Topolog
Hi All,
I would like to start the discussion on KIP-470: TopologyTestDriver test
input and output usability improvements:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
This KIP is inspired by the Discussion in KIP
Github user igalic closed the pull request at:
https://github.com/apache/kafka/pull/1701
---
PartitionBytesInRate: Equivalent to BytesInPerSec, but at a partition level
(i.e. total traffic - throttled and not throttled). This is required for
estimating how long a rebalance will take to complete. B/s. See usability
section below.
SumReplicaLag: This is the sum of all replica lag values on the broker
Ben Stopford created KAFKA-4179:
---
Summary: Replication Throttling: Add Usability Metrics
PartitionBytesInRate & SumReplicaLag
Key: KAFKA-4179
URL: https://issues.apache.org/jira/browse/KAFKA-
GitHub user igalic opened a pull request:
https://github.com/apache/kafka/pull/1701
MINOR: increase usability of shell-scripts
we do this by ensuring that the --zookeeper parameter consistently
defaults to localhost:2181 â as it already does in some places.
You can merge this
Hey guys,
It would be good to tag any JIRA for something which is an confusing or
annoying with the "usability" tag. I am trying to get a list of all these
together so we can take a wack at some of them in a co-ordinated way.
-Jay
Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/jira/browse/KAFKA-990
> Project: Kafka
> Issue Type: Bug
>Repo
le a separate JIRA for Swapnil's
suggestion #27
> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/jira/browse/KAFKA-990
>
rged addpartition to trunk yet. We plan to do once we
merge the 0.8 code to trunk.
> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/ji
t is already done or has
failed. We can try to optimize that in trunk.
> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/ji
[
https://issues.apache.org/jira/browse/KAFKA-990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sriram Subramanian updated KAFKA-990:
-
Attachment: KAFKA-990-v3.patch
> Fix ReassignPartitionCommand and improve usabil
#x27;s 16, currently partition id 0 is also
used for AddPartitionCommand. Shall we change that also?
> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https:
e it mandatory. It is not required when explicit list is
specified. In the case when only topics are specified we do make it mandatory.
31. There is already a tool for that. It is called CheckReassignmentStatus.
> Fix ReassignPartitionCommand and improve us
ents have completed or
the set of partitions that are remaining.
> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/jira/brow
artitions have already been reassigned and
which have not been in initiateReassignPartitionForTopic.
> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https
ends happen on the follower that reduces the sudden influx of messages at
the follower and fetch requests at the leader. Both of these are outside the
scope of this JIRA.
> Fix ReassignPartitionCommand and improve us
[
https://issues.apache.org/jira/browse/KAFKA-990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sriram Subramanian updated KAFKA-990:
-
Attachment: KAFKA-990-v2.patch
> Fix ReassignPartitionCommand and improve usabil
gt; Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/jira/browse/KAFKA-990
> Project: Kafka
> Issue Type: Bug
>
0.8 HEAD
> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/jira/browse/KAFKA-990
> Project: Kafka
>
2 lines).
patching file core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala
Hunk #1 FAILED at 29.
Hunk #2 FAILED at 81.
was (Author: swapnilghike):
The rebased patch also failed for me on 0.8 HEAD
> Fix ReassignPartitionCommand and improve
a fetches. This is a good idea. Can we file a separate JIRA
for this?
> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.o
her words, the user
has to add another option to disable dry run.
3. Since the reassignment process requires fetching old data and may pollute
the pagecache, do you see any performance impact to produce/fetch request
latency when the reassignment is in progress?
> Fix Reas
s is required because a controlled
shutdown attempt may
fail - if there are no other brokers in ISR for a partition led by the broker
being shutdown. In this case we
would want to proceed with a retry (if there are retries left).
> Fix ReassignPartitionCommand and improve us
ough: in shutdownBroker, should we
check liveOrShuttingDownBrokerIds or liveBrokerIds? I saw going back and forth,
and hence a little confused which criteria should be applied here?
> Fix ReassignPartitionCommand and improve us
es I was referring to were in
v1.
+1 on the rebased patch - we can fix the minor comments either on check-in or
in a separate jira.
> Fix ReassignPartitionCommand and improve usability
> --
>
>
atch. I'll review this again this
weekend.
> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/jira/browse/KAFKA-990
>
is a long-running operation.
The dry-run option reduces the need for this but nevertheless I think it's a
good feature to support in the future.
> Fix ReassignPartitionCommand and improve usability
> --
>
>
88: use head instead of assuming 0 exists (start partition id could be
!= 0)
I did not finish going through all the changes in controller, but thought I
would put in my comments so far :)
> Fix ReassignPartitionCo
and hence restarts
the reassignment correctly. This is however not true if we hit #1, which is a
real issue.
> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
>
ence restarts
the reassignment correctly. This is however not true if we hit #1, which is a
real issue.
> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
>
ove usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/jira/browse/KAFKA-990
> Project: Kafka
> Issue Type: Bug
>Reporter: Sriram Subramanian
>
the compilation errors though?
> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/jira/browse/KAFKA-990
> Project:
[
https://issues.apache.org/jira/browse/KAFKA-990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sriram Subramanian updated KAFKA-990:
-
Attachment: KAFKA-990-v1.patch
> Fix ReassignPartitionCommand and improve usabil
Sriram Subramanian created KAFKA-990:
Summary: Fix ReassignPartitionCommand and improve usability
Key: KAFKA-990
URL: https://issues.apache.org/jira/browse/KAFKA-990
Project: Kafka
Issue
72 matches
Mail list logo
