Re: Elasticsearch InputFormat

2018-09-11 Thread Till Rohrmann
Great to hear that it works :-) On Tue, Sep 11, 2018 at 9:28 AM Michael Gendelman wrote: > Hi Till, > > Thanks for the great suggestion! > Seems like it does the job. Here is a sample of the code: > > public class FlinkMain { > public static void main(String[] args) throws Exception { >

Re: Elasticsearch InputFormat

2018-09-11 Thread Michael Gendelman
Hi Till, Thanks for the great suggestion! Seems like it does the job. Here is a sample of the code: public class FlinkMain { public static void main(String[] args) throws Exception { ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); EsInputFormat kvEs

Re: Elasticsearch InputFormat

2018-09-09 Thread Till Rohrmann
Hi Michael, have you considered trying out the EsInputFormat [1] with Flink's HadoopInputFormatBase? That way reading from ElasticSearch might already work out of the box. If not, then adding a dedicated ElasticSearch input format would definitely be helpful. [1] https://github.com/elastic/elasti

Elasticsearch InputFormat

2018-09-08 Thread Michael Gendelman
Hi all, I have a workload where I need to read and transform large amounts of data from Elasticsearch. I'm currently using Flink only for streaming but I though that it can also be a good fit for this kind of batch job. However, I did not find a way to load data from Elasticsearch in parallel to F