Re: [DISCUSS] PIP 193 : Sink preprocessing Function

This may be a matter of personal opinion but I don't see connectors as more self contained than functions. The problem I see in looking at this from a Function point of view is that a Function could have both its Sink and Source not connected to Pulsar which feels awkward. I understand that functio

Re: [DISCUSS] PIP 193 : Sink preprocessing Function

To my understanding, the Pulsar IO Connectors (i.e. Sources/Sinks) are quite self-contained. They move data around. If we want to enable functionality described inside the PIP (process -> write to otherplace), can we think in another way -- allow flexible configuring of a Pulsar Function? Origina

Re: [DISCUSS] PIP 193 : Sink preprocessing Function

Thanks for the feedback Jerry. We don't modify the way sources, sinks and functions are detected when it's based on their fields. The proposal is just to modify the classname of the function applied in the instance so the same detection rules apply. The only difference is when detecting if the sink

Re: [DISCUSS] PIP 193 : Sink preprocessing Function

When the users registers a Sink, the validation code will check that the return type of the Function is a Record. The Function can then be used either individually or together with a Sink. IMO, adding a new API would only add complexity. Le lun. 25 juil. 2022 à 14:55, Baodi Shi a écrit : > > Can

Re: [DISCUSS] PIP 193 : Sink preprocessing Function

My feedback is to make this change as self contained as possible. Can we just have a special implementation of a sink that will run the logic of the "preprocess" function? There are many places in the code where we figure out if it is a source, sink or a function based on the fields in the Functi

Re: [DISCUSS] PIP 193 : Sink preprocessing Function

> Can you explain more what you mean ? This PIP doesn't change the API of a Function and it's already possible to write a Function>. And when declaring a Sink with a Function we'll check that it's the case. I mean: we should constrain the function interface, otherwise, the user may return a struc

Re: [DISCUSS] PIP 193 : Sink preprocessing Function

Thanks for the feedback Asaf > >- preprocess-function: the preprocess function applied before the > >Sink. Starts by builtin:// for built-in functions, function:// for > >package function, http:// or file:// > > > > 1. While this function is applied only before sink? I thought it repl

Re: [DISCUSS] PIP 193 : Sink preprocessing Function

Thanks for your comments Baodi > This proposal looks good to me. > > > Only Functions returning a Record will be authorized to ensure that the > Function sets the Schema explicitly. > > Does this mean that the function return type is fixed Record? Yes > Can the interface declaration of the fu

Re: [DISCUSS] PIP 193 : Sink preprocessing Function

Hi, Few questions: >- preprocess-function: the preprocess function applied before the >Sink. Starts by builtin:// for built-in functions, function:// for >package function, http:// or file:// > > 1. While this function is applied only before sink? I thought it replaces the identity f

Re: [DISCUSS] PIP 193 : Sink preprocessing Function

This proposal looks good to me. > Only Functions returning a Record will be authorized to ensure that the > Function sets the Schema explicitly. Does this mean that the function return type is fixed Record? Can the interface declaration of the function be displayed in the API changes? Thanks

[DISCUSS] PIP 193 : Sink preprocessing Function

Dear Pulsar dev community, I would like to open a discussion here about PIP 193 : Sink preprocessing Function . Best regards Christophe ## Motivation Pulsar IO connectors make it possible to connect Pulsar to an external system: * A Source reads c