In my mind the user API could run everywhere but the simplest thing is to make them available in the Job Manager (where the other REST API lives). They could become a very simple but powerful way to add valuable services to Flink without adding useless complexity to the overall architecture for just a few methods. I don't know whether Spark or Beam allow you to do something like that but IMHO it's super useful (especially from a maintenance point of view wrt the overall architecture complexity).
@Oytun indeed we'd like to avoid recompiling everything when a single user class (i.e. not related to Flink classes) is modified or added. Glad to see that there are other people having the same problem here On Tue, Apr 21, 2020 at 4:39 PM Jeff Zhang <zjf...@gmail.com> wrote: > Hi Flavio, > > I am curious know where service run, Do you create this service in UDF and > run it in TM ? > > Flavio Pompermaier <pomperma...@okkam.it> 于2020年4月21日周二 下午8:30写道： > >> Hi to all, >> many times it happens that we use Flink as a broker towards the data >> layer but we need to be able to get some specific info from the >> data sources we use (i.e. get triggers and relationships from jdbc). >> The quick and dirty way of achieving this is to run a Flink job that >> calls another service to store the required info. Another solution could be >> to add a custom REST service that contains a lot of dependencies already >> provided by Flink, with the risk of having misaligned versions between the >> 2.. >> It would be much simpler to enable users to add custom REST services to >> Flink in a configurable file. something like: >> /myservice1/* -> com.example.MyRestService1 >> /myservice2/* -> com.example.MyRestService2 >> >> The listed classes should be contained in a jar within the Flink lib dir >> and should implement a common interface. >> In order to avoid path collisions with already existing FLINK services, >> the configured path can be further prefixed with some other token (e.g. >> /userapi/*). >> >> What do you think about this? Does it sound reasonable to you? >> Am I the only one that thinks this could be useful for many use cases? >> >> Best, >> Flavio >> > > > -- > Best Regards > > Jeff Zhang >
