Re: Reading csv-files in parallel

2018-05-09 Thread Fabian Hueske
; Best, Esa > > > > *From:* Fabian Hueske > *Sent:* Tuesday, May 8, 2018 10:26 PM > > *To:* Esa Heikkinen > *Cc:* user@flink.apache.org > *Subject:* Re: Reading csv-files in parallel > > > > Hi, > > the Table API / SQL and the DataSet API can be used

RE: Reading csv-files in parallel

2018-05-09 Thread Esa Heikkinen
Hi Sorry the stupid question, but how to connect readTextFile (or readCsvFile), MapFunction and SQL together in Scala code ? Best, Esa From: Fabian Hueske Sent: Tuesday, May 8, 2018 10:26 PM To: Esa Heikkinen Cc: user@flink.apache.org Subject: Re: Reading csv-files in parallel Hi, the Table

Re: Reading csv-files in parallel

2018-05-08 Thread Fabian Hueske
> > *From:* Fabian Hueske > *Sent:* Tuesday, May 8, 2018 2:00 PM > > *To:* Esa Heikkinen > *Cc:* user@flink.apache.org > *Subject:* Re: Reading csv-files in parallel > > > > Hi, > > the easiest approach is to read the CSV files linewise as regular text &g

RE: Reading csv-files in parallel

2018-05-08 Thread Esa Heikkinen
(state-machine-based) logic for reading csv-files by certain order. Esa From: Fabian Hueske Sent: Tuesday, May 8, 2018 2:00 PM To: Esa Heikkinen Cc: user@flink.apache.org Subject: Re: Reading csv-files in parallel Hi, the easiest approach is to read the CSV files linewise as regular text files

Re: Reading csv-files in parallel

2018-05-08 Thread Fabian Hueske
t; > > > Best, Esa > > > > *From:* Fabian Hueske > *Sent:* Monday, May 7, 2018 3:48 PM > *To:* Esa Heikkinen > *Cc:* user@flink.apache.org > *Subject:* Re: Reading csv-files in parallel > > > > Hi Esa, > > you can certainly read CSV files in p

RE: Reading csv-files in parallel

2018-05-08 Thread Esa Heikkinen
To: Esa Heikkinen Cc: user@flink.apache.org Subject: Re: Reading csv-files in parallel Hi Esa, you can certainly read CSV files in parallel. This works very well in a batch query. For streaming queries, that expect data to be ingested in timestamp order this is much more challenging, because you

Re: Reading csv-files in parallel

2018-05-07 Thread Fabian Hueske
Hi Esa, you can certainly read CSV files in parallel. This works very well in a batch query. For streaming queries, that expect data to be ingested in timestamp order this is much more challenging, because you need 1) read the files in the right order and 2) cannot split files (unless you guarante

Reading csv-files in parallel

2018-05-07 Thread Esa Heikkinen
Hi I would want to read many different type csv-files (time series data) parallel using by CsvTableSource. Is that possible in Flink application ? If yes, are there exist the examples about that ? If it is not, do you have any advices how to do that ? Should I combine all csv-files to one csv-