ETL with changing reference data

2017-09-10 Thread Peter Lappo
hi, We are building an ETL style application in Flink that consumes records from a file or a message bus as a DataStream. We are transforming records using SQL and UDFs. The UDF loads reference data in the open method and currently the data loaded remains in memory until the job is cancelled. Th

Re: ETL with changing reference data

2017-09-11 Thread Peter Lappo
eline capture those events with late arriving > event handling might interact with external storage and amend results. > > Thanks, > Chen > > From: Peter Lappo <mailto:pe...@systematicmethods.com> > Sent: Sunday, September 10, 2017 3:00 PM > To: user@flink.apache.org <