Hi, Thanks for your comments. I will refer the overall approach as rich functions until we find a better name.
I think there are some pros and cons of the approach you described. Pros is that it is simple, has clear boundaries, avoids misunderstanding of term "function". So you propose sth like: KStream.valueMapper (ValueMapper vm, RecordContext rc) or having rich functions with only a single init(RecordContext rc) method. Cons is that: - This will bring another set of overloads (if we use RecordContext as a separate parameter). We should consider that the rich functions will be for all main interfaces. - I don't think that we need lambdas in rich functions. It is by definition "rich" so, no single method in interface -> as a result no lambdas. - I disagree that rich functions should only contain init() method. This depends on each interface. For example, for specific interfaces we can add methods (like punctuate()) to their rich functions. Cheers, Jeyhun On Thu, May 25, 2017 at 1:02 AM Matthias J. Sax <matth...@confluent.io> wrote: > I confess, the term is borrowed from Flink :) > > Personally, I never thought about it, but I tend to agree with Michal. I > also want to clarify, that the main purpose is the ability to access > record metadata. Thus, it might even be sufficient to only have "init". > > An alternative would of course be, to pass in the RecordContext as > method parameter. This would allow us to drop "init()". This might even > allow to use Lambdas and we could keep the name RichFunction as we > preserve the nature of being a function. > > > -Matthias > > On 5/24/17 12:13 PM, Jeyhun Karimov wrote: > > Hi Michal, > > > > Thanks for your comments. I see your point and I agree with it. However, > > I don't have a better idea for naming. I checked MR source code. There > > it is used JobConfigurable and Closable, two different interfaces. Maybe > > we can rename RichFunction as Configurable? > > > > > > Cheers, > > Jeyhun > > > > On Tue, May 23, 2017 at 2:58 PM Michal Borowiecki > > <michal.borowie...@openbet.com <mailto:michal.borowie...@openbet.com>> > > wrote: > > > > Hi Jeyhun, > > > > I understand your argument about "Rich" in RichFunctions. Perhaps > > I'm just being too puritan here, but let me ask this anyway: > > > > What is it that makes something a function? To me a function is > > something that takes zero or more arguments and possibly returns a > > value and while it may have side-effects (as opposed to "pure > > functions" which can't), it doesn't have any life-cycle of its own. > > This is what, in my mind, distinguishes the concept of a "function" > > from that of more vaguely defined concepts. > > > > So if we add a life-cycle to a function, in that understanding, it > > doesn't become a rich function but instead stops being a function > > altogether. > > > > You could say it's "just semantics" but to me precise use of > > language in the given context is an important foundation for good > > engineering. And in the context of programming "function" has a > > precise meaning. Of course we can say that in the context of Kafka > > Streams "function" has a different, looser meaning but I'd argue > > that won't do anyone any good. > > > > On the other hand other frameworks such as Flink use this > > terminology, so it could be that consistency is the reason. I'm > > guessing that's why the name was proposed in the first place. My > > point is simply that it's a poor choice of wording and Kafka Streams > > don't have to follow that to the letter. > > > > Cheers, > > > > Michal > > > > > > On 23/05/17 13:26, Jeyhun Karimov wrote: > >> Hi Michal, > >> > >> Thanks for your comments. > >> > >> > >> To me at least it feels strange that something is called a > >> function yet doesn't follow the functional interface > >> definition of having just one abstract method. I suppose init > >> and close could be made default methods with empty bodies once > >> Java 7 support is dropped to mitigate that concern. Still, I > >> feel some resistance to consider something that requires > >> initialisation and closing (which implies holding state) as > >> being a function. Sounds more like the Processor/Transformer > >> kind of thing semantically, rather than a function. > >> > >> > >> - If we called the interface name only Function your assumptions > >> will hold. However, the keyword Rich by definition implies that we > >> have a function (as you described, with one abstract method and > >> etc) but it is rich. So, there are multiple methods in it. > >> Ideally it should be: > >> > >> public interface RichFunction extends Function { // this > >> is the Function that you described > >> void close(); > >> void init(Some params); > >> ... > >> } > >> > >> > >> The KIP says there are multiple use-cases for this but doesn't > >> enumerate any - I think some examples would be useful, > >> otherwise that section sounds a little bit vague. > >> > >> > >> I thought it is obvious by definition but I will update it. Thanks. > >> > >> > >> IMHO, it's the access to the RecordContext is where the added > >> value lies but maybe I'm just lacking in imagination, so I'm > >> asking all this to better understand the rationale for init() > >> and close(). > >> > >> > >> Maybe I should add some examples. Thanks. > >> > >> > >> Cheers, > >> Jeyhun > >> > >> On Mon, May 22, 2017 at 11:02 AM, Michal Borowiecki > >> <michal.borowie...@openbet.com > >> <mailto:michal.borowie...@openbet.com>> wrote: > >> > >> Hi Jeyhun, > >> > >> I'd like to understand better the premise of RichFunctions and > >> why |init(Some params)|,| close() |are said to be needed. > >> > >> To me at least it feels strange that something is called a > >> function yet doesn't follow the functional interface > >> definition of having just one abstract method. I suppose init > >> and close could be made default methods with empty bodies once > >> Java 7 support is dropped to mitigate that concern. Still, I > >> feel some resistance to consider something that requires > >> initialisation and closing (which implies holding state) as > >> being a function. Sounds more like the Processor/Transformer > >> kind of thing semantically, rather than a function. > >> > >> The KIP says there are multiple use-cases for this but doesn't > >> enumerate any - I think some examples would be useful, > >> otherwise that section sounds a little bit vague. > >> > >> IMHO, it's the access to the RecordContext is where the added > >> value lies but maybe I'm just lacking in imagination, so I'm > >> asking all this to better understand the rationale for init() > >> and close(). > >> > >> Thanks, > >> Michał > >> > >> On 20/05/17 17:05, Jeyhun Karimov wrote: > >>> Dear community, > >>> > >>> As we discussed in KIP-149 [DISCUSS] thread [1], I would like > to initiate > >>> KIP for rich functions (interfaces) [2]. > >>> I would like to get your comments. > >>> > >>> > >>> [1] > >>> > http://search-hadoop.com/m/Kafka/uyzND1PMjdk2CslH12?subj=Re+DISCUSS+KIP+149+Enabling+key+access+in+ValueTransformer+ValueMapper+and+ValueJoiner > >>> [2] > >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-159%3A+Introducing+Rich+functions+to+Streams > >>> > >>> > >>> Cheers, > >>> Jeyhun > >> -- > >> <http://www.openbet.com/> Michal Borowiecki > >> Senior Software Engineer L4 > >> T: +44 208 742 1600 <+44%2020%208742%201600> > <tel:+44%2020%208742%201600> > >> +44 203 249 8448 <+44%2020%203249%208448> > <tel:+44%2020%203249%208448> > >> > >> E: michal.borowie...@openbet.com > >> <mailto:michal.borowie...@openbet.com> > >> W: www.openbet.com <http://www.openbet.com/> > >> > >> > >> OpenBet Ltd > >> Chiswick Park Building 9 > >> 566 Chiswick High Rd > >> London > >> W4 5XT > >> UK > >> > >> > >> <https://www.openbet.com/email_promo> > >> > >> This message is confidential and intended only for the > >> addressee. If you have received this message in error, please > >> immediately notify the postmas...@openbet.com > >> <mailto:postmas...@openbet.com> and delete it from your system > >> as well as any copies. The content of e-mails as well as > >> traffic data may be monitored by OpenBet for employment and > >> security purposes. To protect the environment please do not > >> print this e-mail unless necessary. OpenBet Ltd. Registered > >> Office: Chiswick Park Building 9, 566 Chiswick High Road, > >> London, W4 5XT, United Kingdom. A company registered in > >> England and Wales. Registered no. 3134634. VAT no. GB927523612 > >> > > -- > > <http://www.openbet.com/> Michal Borowiecki > > Senior Software Engineer L4 > > T: +44 208 742 1600 <+44%2020%208742%201600> > <tel:+44%2020%208742%201600> > > +44 203 249 8448 <+44%2020%203249%208448> > <tel:+44%2020%203249%208448> > > > > E: michal.borowie...@openbet.com > > <mailto:michal.borowie...@openbet.com> > > W: www.openbet.com <http://www.openbet.com/> > > > > > > OpenBet Ltd > > Chiswick Park Building 9 > > 566 Chiswick High Rd > > London > > W4 5XT > > UK > > > > > > <https://www.openbet.com/email_promo> > > > > This message is confidential and intended only for the addressee. If > > you have received this message in error, please immediately notify > > the postmas...@openbet.com <mailto:postmas...@openbet.com> and > > delete it from your system as well as any copies. The content of > > e-mails as well as traffic data may be monitored by OpenBet for > > employment and security purposes. To protect the environment please > > do not print this e-mail unless necessary. OpenBet Ltd. Registered > > Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 > > 5XT, United Kingdom. A company registered in England and Wales. > > Registered no. 3134634. VAT no. GB927523612 > > > > -- > > -Cheers > > > > Jeyhun > > -- -Cheers Jeyhun
