I know some users do the same thing in spark. Usually the service run spark driver side. But flink is different from spark. Spark driver is equal to flink client + flink job manager. I don't think currently we allow to run any user code in job manager. So allow running user defined service in job manager might a big change for flink.
Flavio Pompermaier <pomperma...@okkam.it> 于2020年4月21日周二 下午11:06写道： > In my mind the user API could run everywhere but the simplest thing is to > make them available in the Job Manager (where the other REST API lives). > They could become a very simple but powerful way to add valuable services > to Flink without adding useless complexity to the overall architecture for > just a few methods. > I don't know whether Spark or Beam allow you to do something like that but > IMHO it's super useful (especially from a maintenance point of view wrt the > overall architecture complexity). > > @Oytun indeed we'd like to avoid recompiling everything when a single user > class (i.e. not related to Flink classes) is modified or added. Glad to see > that there are other people having the same problem here > > On Tue, Apr 21, 2020 at 4:39 PM Jeff Zhang <zjf...@gmail.com> wrote: > >> Hi Flavio, >> >> I am curious know where service run, Do you create this service in UDF >> and run it in TM ? >> >> Flavio Pompermaier <pomperma...@okkam.it> 于2020年4月21日周二 下午8:30写道： >> >>> Hi to all, >>> many times it happens that we use Flink as a broker towards the data >>> layer but we need to be able to get some specific info from the >>> data sources we use (i.e. get triggers and relationships from jdbc). >>> The quick and dirty way of achieving this is to run a Flink job that >>> calls another service to store the required info. Another solution could be >>> to add a custom REST service that contains a lot of dependencies already >>> provided by Flink, with the risk of having misaligned versions between the >>> 2.. >>> It would be much simpler to enable users to add custom REST services to >>> Flink in a configurable file. something like: >>> /myservice1/* -> com.example.MyRestService1 >>> /myservice2/* -> com.example.MyRestService2 >>> >>> The listed classes should be contained in a jar within the Flink lib dir >>> and should implement a common interface. >>> In order to avoid path collisions with already existing FLINK services, >>> the configured path can be further prefixed with some other token (e.g. >>> /userapi/*). >>> >>> What do you think about this? Does it sound reasonable to you? >>> Am I the only one that thinks this could be useful for many use cases? >>> >>> Best, >>> Flavio >>> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > -- Best Regards Jeff Zhang
