Hi all! In general, nice idea to support this integration with Atlas.
I think we could make this a bit easier/lightweight with a small change. One of the issues that is not super nice is that this starts exposing the (currently empty) Pipeline interface in the public API. The Pipeline is an SPI interface that would be good to hide in the API. Since 1.10, Flink has the notion of Executors, which take the pipeline and execute them. Meaning each pipeline is passed on anyways. And executors are already configurable in the Flink configuration. So, instead of passing the pipeline both "down" (to the executor) and "to the side" (JobListener), could we just have a wrapping "AtlasExecutor" that takes the pipeline, does whatever it wants, and then passes it to the proper executor? This would also have the advantage that it supports making changes to the pipeline, if needed in the future. For example, if there is ever the need to add additional configuration fields, set properties, add "labels" or so, this could be easily done in the suggested approach. I tried to sketch this in the picture below, pardon my bad drawing. [image: Listener_Executor.png] https://xjcrkw.bn.files.1drv.com/y4pWH57aEvLU5Ww4REC9XLi7nJMLGHq2smPSzaslU8ogywFDcMkP-_Rsl8B1njf4qphodim6bgnLTNFwNoEuwFdTuA2Xmf7CJ_8lTJjrKlFlDwrugVeBQzEhAY7n_5j2bumwDBf29jn_tZ1ueZxe2slhLkPC-9K6Dry_vpvRvZRY-CSnQXxj9jDf7P53Vz1K9Ez/Listener_Executor.png?psid=1 Best, Stephan On Wed, Mar 11, 2020 at 11:41 AM Aljoscha Krettek <aljos...@apache.org> wrote: > Thanks! I'm reading the document now and will get back to you. > > Best, > Aljoscha >
