What is the status of this KIP? -Matthias
On 2/13/18 1:43 PM, Matthias J. Sax wrote:
> Is there any update for this KIP?
>
>
> -Matthias
>
> On 12/4/17 2:08 PM, Matthias J. Sax wrote:
>> Jeyhun,
>>
>> thanks for updating the KIP.
>>
>> I am wondering if you intend to add a new class `Produced`? There is
>> already `org.apache.kafka.streams.kstream.Produced`. So if we want to
>> add a new class, it must have a different name -- or we might be able to
>> merge both into one?
>>
>> Also, for the KStream overlaods of `through()` and `to()`, can you add
>> the different behavior using different overloads? It's not clear from
>> the KIP what the semantics are.
>>
>>
>> -Matthias
>>
>> On 11/17/17 3:27 PM, Jeyhun Karimov wrote:
>>> Hi,
>>>
>>> Thanks for your comments. I agree with Matthias partially.
>>> I think we should relax some requirements related with to() and through()
>>> methods.
>>> IMHO, Produced class can cover (existing/to be created) topic information,
>>> and which will ease our effort:
>>>
>>> KStream.to(Produced topicInfo)
>>> KStream.through(Produced topicInfo)
>>>
>>> This will decrease the number of overloads but we will need to deprecate
>>> the existing to() and through() methods, perhaps.
>>> I updated the KIP accordingly.
>>>
>>>
>>> Cheers,
>>> Jeyhun
>>>
>>> On Thu, Nov 16, 2017 at 10:21 PM Matthias J. Sax <matth...@confluent.io>
>>> wrote:
>>>
>>>> @Jan:
>>>>
>>>> The `Produced` class was introduced in 1.0 to specify key and valud
>>>> Serdes (and partitioner) if data is written into a topic.
>>>>
>>>> Old API:
>>>>
>>>> KStream#to("topic", keySerde, valueSerde);
>>>>
>>>> New API:
>>>>
>>>> KStream#to("topic", Produced.with(keySerde, valueSerde));
>>>>
>>>>
>>>> This allows to reduce the number of overloads for `to()` (and
>>>> `through()` that follows the same pattern) -- the second parameter is
>>>> used to cover all different variations of option parameters users can
>>>> specify, while we only have 2 overload for `to()` itself.
>>>>
>>>> What is still unclear to me it, what you mean by this topic prefix
>>>> thing? Either a user cares about the topic name and thus, must create
>>>> and manage it manually. Or the user does not care, and Streams create
>>>> it. How would this prefix idea fit in here?
>>>>
>>>>
>>>>
>>>> @Guozhang:
>>>>
>>>> My idea was to extend `Produced` with the hint we want to give for
>>>> creating internal topic and pass a optional `Produced` parameter. There
>>>> are multiple things we can do here:
>>>>
>>>> 1) stream.through(null, Produced...).groupBy().aggregate()
>>>> -> just allow for `null` topic name indicating that Streams should
>>>> create an internal topic
>>>>
>>>> 2) stream.through(Produced...).groupBy().aggregate()
>>>> -> add one overload taking an mandatory `Produced`
>>>>
>>>> We use `Serialized` to picky back the information
>>>>
>>>> 3) stream.groupBy(Serialized...).aggregate()
>>>> and stream.groupByKey(Serialized...).aggregate()
>>>> -> we don't need new top level overloads
>>>>
>>>>
>>>> There are different trade-offs for those alternatives and maybe there
>>>> are other ways to change the API. It's just to push the discussion further.
>>>>
>>>>
>>>> -Matthias
>>>>
>>>> On 11/12/17 1:22 PM, Jan Filipiak wrote:
>>>>> Hi Gouzhang,
>>>>>
>>>>> this felt like these questions are supposed to be answered by me.
>>>>> I do not understand the first one. I don't understand why the user
>>>>> shouldn't be able to specify a suffix for the topic name.
>>>>>
>>>>> For the third question I am not 100% familiar if the Produced class
>>>>> came to existence
>>>>> at all. I remember proposing it somewhere in our redo DSL discussion that
>>>>> I dropped out of later. Finally any call that does:
>>>>>
>>>>> 1. create the internal topic
>>>>> 2. register sink
>>>>> 3. register source
>>>>>
>>>>> will always get the work done. If we have a Produced like class. putting
>>>>> all the parameters
>>>>> in there make sense. (Partitioner, serde, PartitionHint, internal, name
>>>>> ... )
>>>>>
>>>>> Hope this helps?
>>>>>
>>>>>
>>>>> On 10.11.2017 07:54, Guozhang Wang wrote:
>>>>>> A few clarification questions on the proposal details.
>>>>>>
>>>>>> 1. API: although the repartition only happens at the final stateful
>>>>>> operations like agg / join, the repartition flag info was actually
>>>> passed
>>>>>> from an earlier operator like map / groupBy. So what should be the new
>>>>>> API
>>>>>> look like? For example, if we do
>>>>>>
>>>>>> stream.groupBy().through("topic-name", Produced..).aggregate
>>>>>>
>>>>>> This would be add a bunch of APIs to GroupedKStream/KTable
>>>>>>
>>>>>> 2. Semantics: as Matthias mentioned, today any topics defined in
>>>>>> "through()" call is considered a user topic, and hence users are
>>>>>> responsible for managing them, including the topic name. For this KIP's
>>>>>> purpose, though, users would not care about the topic name. I.e. as a
>>>>>> user
>>>>>> I still want to make it be an internal topic so that I do not need to
>>>>>> worry
>>>>>> about it at all, but only specify num.partitions.
>>>>>>
>>>>>> 3. Details: in Produced we do not have specs for specifying the
>>>>>> num.partitions or should we repartition or not. So it is still not
>>>>>> clear to
>>>>>> me how we would make use of that to achieve what's in the old
>>>>>> proposal's RepartitionHint class.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Guozhang
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 6, 2017 at 1:21 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>>>>>>
>>>>>>> bq. enlarge the score of through()
>>>>>>>
>>>>>>> I guess you meant scope.
>>>>>>>
>>>>>>> On Mon, Nov 6, 2017 at 1:15 PM, Jeyhun Karimov <je.kari...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Sorry for the late reply. I am convinced that we should enlarge the
>>>>>>>> score
>>>>>>>> of through() (add more overloads) instead of introducing a separate
>>>> set
>>>>>>> of
>>>>>>>> overloads to other methods.
>>>>>>>> I will update the KIP soon based on the discussion and inform.
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Jeyhun
>>>>>>>>
>>>>>>>> On Mon, Nov 6, 2017 at 9:18 PM Jan Filipiak <jan.filip...@trivago.com
>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Sorry for not beeing 100% up to date.
>>>>>>>>> Back then we had the discussion that when an operation puts a >Sink<
>>>>>>>>> into the topology, a >Produced<
>>>>>>>>> parameter is added. This produced parameter could have internal or
>>>>>>>>> external. If internal I think the name would still make
>>>>>>>>> a great suffix for the topic name
>>>>>>>>>
>>>>>>>>> Is this plan still around? Otherwise having the name as suffix is
>>>>>>>>> probably always good it can help the user quicker to identify hot
>>>>>>> topics
>>>>>>>>> that need more
>>>>>>>>> partitions if he has many of these internal repartitions
>>>>>>>>>
>>>>>>>>> Best Jan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 06.11.2017 20:13, Matthias J. Sax wrote:
>>>>>>>>>> I absolute agree with what you say. It's not a requirement to
>>>>>>> specify a
>>>>>>>>>> topic name -- and this was the idea -- if user does specify a name,
>>>>>>> we
>>>>>>>>>> treat as is -- if users does not specify a name, Streams create an
>>>>>>>>>> internal topic.
>>>>>>>>>>
>>>>>>>>>> The goal of the Jira is to allow a simplified way to control
>>>>>>>>>> repartitioning (atm, user needs to manually create a topic and use
>>>>>>> via
>>>>>>>>>> through()).
>>>>>>>>>>
>>>>>>>>>> Thus, the idea is to make the topic name parameter of through
>>>>>>> optional.
>>>>>>>>>> It's of course just an idea. Happy do have a other API design. The
>>>>>>> goal
>>>>>>>>>> was, to avoid to many new overloads.
>>>>>>>>>>
>>>>>>>>>>>> Could you clarify exactly what you mean by keeping the current
>>>>>>>>> distinction?
>>>>>>>>>> Current distinction is: user topics are created manually and user
>>>>>>>>>> specifies the name -- internal topics are created by Kafka Streams
>>>>>>> and
>>>>>>>>>> an name is generated automatically.
>>>>>>>>>>
>>>>>>>>>> -> through("user-topic")
>>>>>>>>>> -> through(TopicConfig.withNumberOfPartitions(5)) // Streams creates
>>>>>>>> an
>>>>>>>>>> internal topic
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Matthias
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 11/6/17 6:56 PM, Thomas Becker wrote:
>>>>>>>>>>> Could you clarify exactly what you mean by keeping the current
>>>>>>>>> distinction?
>>>>>>>>>>> Actually, re-reading the KIP and JIRA, it's not clear that being
>>>>>>> able
>>>>>>>>> to specify a custom name is actually a requirement. If the goal is to
>>>>>>>>> control repartitioning and tune parallelism, maybe we can just
>>>>>>>>> sidestep
>>>>>>>>> this issue altogether by removing the ability to set a different
>>>> name.
>>>>>>>>>>> On Mon, 2017-11-06 at 16:51 +0100, Matthias J. Sax wrote:
>>>>>>>>>>>
>>>>>>>>>>> That's a good point. In current design, we strictly distinguish
>>>>>>> both.
>>>>>>>>>>> For example, the reset tools deletes internal topics (starting with
>>>>>>>>>>> prefix `<application.id>-` and ending with either `-repartition`
>>>> or
>>>>>>>>>>> `-changelog`.
>>>>>>>>>>>
>>>>>>>>>>> Thus, from my point of view, it would make sense to keep the
>>>> current
>>>>>>>>>>> distinction.
>>>>>>>>>>>
>>>>>>>>>>> -Matthias
>>>>>>>>>>>
>>>>>>>>>>> On 11/6/17 4:45 PM, Thomas Becker wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I think this sounds good as well. It's worth clarifying whether
>>>>>>> topics
>>>>>>>>> that are named by the user but created by streams are considered
>>>>>>>> "internal"
>>>>>>>>> topics also.
>>>>>>>>>>> On Sun, 2017-11-05 at 23:02 +0100, Matthias J. Sax wrote:
>>>>>>>>>>>
>>>>>>>>>>> My idea was, to relax the requirement for through() that a topic
>>>>>>> must
>>>>>>>> be
>>>>>>>>>>> created manually before startup.
>>>>>>>>>>>
>>>>>>>>>>> Thus, if no through() call is made, a (internal) topic is created
>>>>>>> the
>>>>>>>>>>> same way we do it currently.
>>>>>>>>>>>
>>>>>>>>>>> If one uses `through(String topicName)` we keep the current
>>>> behavior
>>>>>>>> and
>>>>>>>>>>> require users to create the topic manually.
>>>>>>>>>>>
>>>>>>>>>>> The reasoning is as follows: if a user creates a topic manually, a
>>>>>>>> user
>>>>>>>>>>> can just use it for repartitioning. As the topic is already there,
>>>>>>>> there
>>>>>>>>>>> is no need to specify any topic configs.
>>>>>>>>>>>
>>>>>>>>>>> We add a new `through()` overload (details TBD) that allows to
>>>>>>> specify
>>>>>>>>>>> topic configs and Streams create the topic with those configs.
>>>>>>>>>>>
>>>>>>>>>>> Reasoning: user don't want to manage topic manually, thus, it's
>>>>>>> still
>>>>>>>> an
>>>>>>>>>>> internal topic and Streams create the topic name automatically as
>>>>>>> for
>>>>>>>>>>> all other internal topics. However, users gets some more control
>>>>>>> about
>>>>>>>>>>> topic parameters like number of partitions (we should discuss what
>>>>>>>> other
>>>>>>>>>>> configs would be useful).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Does this make sense?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -Matthias
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 11/5/17 1:21 AM, Jan Filipiak wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Im not 100 % up to date what version 1.0 DSL looks like ATM.
>>>>>>>>>>> I just would argue that repartitioning should be an own API call
>>>>>>> like
>>>>>>>>>>> through or something.
>>>>>>>>>>> One can use through or to already to get this. I would argue one
>>>>>>>> should
>>>>>>>>>>> look there instead of overloads
>>>>>>>>>>>
>>>>>>>>>>> Best Jan
>>>>>>>>>>>
>>>>>>>>>>> On 04.11.2017 16:01, Jeyhun Karimov wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Dear community,
>>>>>>>>>>>
>>>>>>>>>>> I would like to initiate discussion on KIP-221 [1] based on issue
>>>>>>> [2].
>>>>>>>>>>> Please feel free to comment.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>>
>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>> 221%3A+Repartition+Topic+Hints+in+Streams
>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/KAFKA-6037
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Jeyhun
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ________________________________
>>>>>>>>>>>
>>>>>>>>>>> This email and any attachments may contain confidential and
>>>>>>> privileged
>>>>>>>>> material for the sole use of the intended recipient. Any review,
>>>>>>> copying,
>>>>>>>>> or distribution of this email (or any attachments) by others is
>>>>>>>> prohibited.
>>>>>>>>> If you are not the intended recipient, please contact the sender
>>>>>>>>> immediately and permanently delete this email and any attachments. No
>>>>>>>>> employee or agent of TiVo Inc. is authorized to conclude any binding
>>>>>>>>> agreement on behalf of TiVo Inc. by email. Binding agreements with
>>>>>>>>> TiVo
>>>>>>>>> Inc. may only be made by a signed written agreement.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ________________________________
>>>>>>>>>>>
>>>>>>>>>>> This email and any attachments may contain confidential and
>>>>>>> privileged
>>>>>>>>> material for the sole use of the intended recipient. Any review,
>>>>>>> copying,
>>>>>>>>> or distribution of this email (or any attachments) by others is
>>>>>>>> prohibited.
>>>>>>>>> If you are not the intended recipient, please contact the sender
>>>>>>>>> immediately and permanently delete this email and any attachments. No
>>>>>>>>> employee or agent of TiVo Inc. is authorized to conclude any binding
>>>>>>>>> agreement on behalf of TiVo Inc. by email. Binding agreements with
>>>>>>>>> TiVo
>>>>>>>>> Inc. may only be made by a signed written agreement.
>>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
signature.asc
Description: OpenPGP digital signature
