Re: unclear exception when writing to elasticsearch

ow I’m trying your setup right now, and will get back > to you once I confirm the results. > > - Gordon > > > On March 1, 2017 at 9:15:16 PM, Martin Neumann (mneum...@sics.se) wrote: > > I created the project using the maven archetype > <https://ci.apache.org/projects/flink/fli

Re: unclear exception when writing to elasticsearch

link-dist > mvn clean install > > *Note:* To check your Maven version, run mvn --version. > > On Wed, Mar 1, 2017 at 12:19 PM, Martin Neumann wrote: > >> I tried to change the elastic search version to 2.4.1 which results in a >> new ex

Re: unclear exception when writing to elasticsearch

he Elasticsearch client of your > desired version in your project. > > You can also check what Elasticsearch client version the project is using > by checking `mvn dependency:tree` from the base directory of your project. > > Cheers, > Gordon > > > On March 1, 2017 at 1:21

Re: unclear exception when writing to elasticsearch

t; projects/flink/flink-docs-release-1.3/dev/linking.html# > packaging-dependencies-with-your-usercode-with-maven. > > Let me know if the problem remains after trying out the above :-) > > Cheers, > Gordon > > On March 1, 2017 at 12:24:08 AM, Martin Neumann (mneum...@sics.se

unclear exception when writing to elasticsearch

Hej, I'm trying to write to elastic search from a streaming application and I get a weird error message I that I can't decipher. Hopefully, someone here can help me. I'm trying to run the java example

Flink streaming connect and split streams

Hej, I'm currently playing around with some machine learning algorithms in Flink streaming. I have an input stream that I partition by key and then do a map on each of the keys, feeding a model and producing a prediction output. Periodically each operator needs to send model updates to all other

Re: get start and end time stamp from time window

provide only one (the aggregated) element. > > See the apply method on WindowedStream with the following signature: > apply(initialValue: R, foldFunction: FoldFunction[T, R], function: > WindowFunction[R, R, K, W]): DataStream[R] > > Cheers, Fabian > > 2016-05-11 20:16 GMT+02

Re: Local Cluster have problem with connect to elasticsearch

Hi, Are you sure the elastic cluster is running correctly? Open a browser and try 127.0.0.1:9200 that should give you the overview of the cluster. If you don't get it there is something wrong with the setup. Its also a good way to double check the cluster.name (I got that wrong more than once) I

get start and end time stamp from time window

Hej, I have a windowed stream and I want to run a (generic) fold function on it. The result should have the start and the end time stamp of the window as fields (so I can relate it to the original data). *Is there a simple way to get the timestamps from within the fold function?* I could find the

Re: TypeVariable problems

Aljoscha > > On Fri, 29 Apr 2016 at 12:25 Martin Neumann wrote: > >> Hej, >> >> I have a construct of different generic classes stacked on each other to >> create a library (so the type variables get handed on). And I have some >> trouble getting it to

Fwd: TypeVariable problems

Hej, I have a construct of different generic classes stacked on each other to create a library (so the type variables get handed on). And I have some trouble getting it to work. The current offender is a Class with 3 type variables internally it calls: .fold(new Tuple3<>(keyInit ,new Tuple2(0d,0

Re: Help with generics

l.class). If > that doesn't work you can try and pass in a function that gives you a > TypeInformation for your model type M. > > Cheers, > Aljoscha > > On Thu, 21 Apr 2016 at 14:16 Martin Neumann wrote: > >> Hej, >> >> I pass an instance of M in the constru

Re: Help with generics

an do is manually pass a > TypeInformation or a TypeSerializer to the AnomalyFlatMap and use > that when creating the state descriptor. > > Cheers, > Aljoscha > > On Thu, 21 Apr 2016 at 13:45 Martin Neumann wrote: > >> Hey, >> >> I have a FlatMap that uses some gen

Help with generics

Hey, I have a FlatMap that uses some generics (appended at the end of the mail). I have some trouble with the type inference running into InvalidTypesException on the first line in the open function. How can I fix it? Cheers Martin public class AnomalyFlatMap extends RichFlatMapFunction, Tup

Re: streaming hdfs sub folders

r. cheers Martin On Fri, Feb 19, 2016 at 2:39 PM, Robert Metzger wrote: > Hi Martin, > > where is the null pointer exception thrown? > I think you didn't call the open() method of the AvroInputFormat. Maybe > that's the issue. > > On Thu, Feb 18, 2016 at 5:01 PM, Martin

Re: streaming hdfs sub folders

Pq()); if (res != null) collector.collect(res); } } } On Thu, Feb 18, 2016 at 4:06 PM, Martin Neumann wrote: > I guess I need to set the parallelism for the FlatMap to 1 to make sure I > read one file at a time. The downside I see with this is that I will be no

Re: streaming hdfs sub folders

eam inStream = >> env.readFile(new AvroInputFormat(new >> Path(filePath), EndSongCleanedPq.class), filePath); >> >> >> On Wed, Feb 17, 2016 at 7:33 PM, Martin Neumann wrote: >> >>> The program is a DataStream program, it usually it gets the data from &

Re: streaming hdfs sub folders

16 at 7:33 PM, Martin Neumann wrote: > The program is a DataStream program, it usually it gets the data from > kafka. It's an anomaly detection program that learns from the stream > itself. The reason I want to read from files is to test different settings > of the algorithm a

Re: streaming hdfs sub folders

If you still want parallel reads, that means you need to sync at > some point, wait for all parallel parts to finish with the "00" work before > anyone may start with the "01" work. > > Is your training program a DataStream or a DataSet program?` > > Stephan

streaming hdfs sub folders

Hi, I have a streaming machine learning job that usually runs with input from kafka. To tweak the models I need to run on some old data from HDFS. Unfortunately the data on HDFS is spread out over several subfolders. Basically I have a datum with one subfolder for each hour within those are the a

Re: streaming using DeserializationSchema

it later. cheers Martin On Fri, Feb 12, 2016 at 10:47 PM, Nick Dimiduk wrote: > My input file contains newline-delimited JSON records, one per text line. > The records on the Kafka topic are JSON blobs encoded to UTF8 and written > as bytes. > > On Fri, Feb 12, 2016 at 1:41 PM,

Re: streaming using DeserializationSchema

s the "flow" application, so I pass it into the factory method. > Tweak the ParameterTools options for the "job" application, et voilà! > > Sorry I don't have example code for you; this would be a good example to > contribute back to the community's examp

Re: streaming using DeserializationSchema

topic. This also has the > additional benefit that you are actually testing against kafka. > > Cheers, > Gyula > > Martin Neumann ezt írta (időpont: 2016. febr. 12., P, > 0:20): > >> Hej, >> >> I have a stream program reading data from Kafka where the data

streaming using DeserializationSchema

Hej, I have a stream program reading data from Kafka where the data is in avro. I have my own DeserializationSchema to deal with it. For testing reasons I want to read a dump from hdfs instead, is there a way to use the same DeserializationSchema to read from an avro file stored on hdfs? cheers

elasticsearch connection trouble

Hi everyone, I'm trying to connect my flink streaming job to elastic search but I have trouble to make it work. Here is the config I'm using for the connector: HashMap elConf = new HashMap<>(); elConf.put("bulk.flush.max.actions", "1"); elConf.put("cluster.name", "logelask"); List transports = ne

Re: Streaming time window

> Collector out) { > // YOUR CODE > window.getEnd() > } > }) > > Best, Fabian > > 2015-12-10 14:41 GMT+01:00 Martin Neumann : > >> Hi Fabian, >> >> thanks for your answer. Can I do the same in java using normal time >>

Re: Streaming time window

> out: Collector[(Int, Long, Int)]) => > out.collect( ( key, window.getEnd, vals.map( _._2 ).sum ) ) > } > > Cheers, Fabian > > 2015-12-10 12:04 GMT+01:00 Martin Neumann : > >> Hej, >> >> Is it possible to extract the start and end window time stamps fro

Streaming time window

Hej, Is it possible to extract the start and end window time stamps from within a window operator? I have an event time based window that does a simple fold function. I want to put the output into elasticsearch and want to preserve the start and end timestamp of the data so I can directly compare

Re: How to preserve KeyedDataStream

ction) > > here, the ReduceFunction does the WordCount-like counting while in the > WindowFunction you get the final result and store it inside your model. > > Let me know if you need more information. > > Cheers, > Aljoscha > > On 03 Nov 2015, at 11:28, Martin Neumann wr

How to preserve KeyedDataStream

Hej, I want to do the following thing: 1. Split a Stream of incoming Logs by host address. 2. For each Key, create time based windows 3. Count the number of items in the window 4. Feed it into a statistical model that is maintained for each host Since I don't have fields to sum upon, I use a (win

Flink Streaming and Google Cloud Pub/Sub?

Hej, Has anyone tried use connect Flink Streaming to Google Cloud Pub/Sub and has a code example for me? If I have to implement my own sources and sinks are there any good tutorials for that? cheers Martin

flink with kafka 0.7

Hej, I want to connect Flink streaming to a Kafka 0.7 cluster. Will this work with the latest release or does the Kafka implementation rely on kafka 0.8? cheers Martin

Re: Event time in Flink streaming

Hej, I want to give TimeTriggerPolicy a try and see how much of a problem it will be in this use case. Is there any example on how to use it? I looked at the API descriptions but I'm confused now. cheers Martin On Fri, Aug 28, 2015 at 5:35 PM, Martin Neumann wrote: > The stream con

Re: Performance Issue

Hej, This sounds like it could be a garbage collection problem. Do you instantiate any classes inside any of the operators (e.g. in the KeySelector). You can also try to run it locally and use something like jstat to rule this out. cheers Martin On Mon, Sep 7, 2015 at 12:00 PM, Rico Bergmann wr

Re: Splitting Streams

;> could grouping be a solution to your problem? >> >> Cheers, >> Till >> >> On Thu, Sep 3, 2015 at 11:56 AM, Martin Neumann wrote: >> >>> Hej, >>> >>> I have a Stream of json objects of several different types. I want to >>> spl

Splitting Streams

Hej, I have a Stream of json objects of several different types. I want to split this stream into several streams each of them dealing with one type. (so its not partitioning) The only Way I found so far is writing a bunch of filters and connect them to the source directly. This way I will have a

Re: Event time in Flink streaming

; > On Fri, 28 Aug 2015 at 16:11 Matthias J. Sax < > mj...@informatik.hu-berlin.de> wrote: > >> Hi Martin, >> >> you need to implement you own policy. However, this should be be >> complicated. Have a look at "TimeTriggerPolicy". You just need to >

Event time in Flink streaming

Hej, I have a stream of timestamped events I want to process in Flink streaming. Di I have to write my own policies to do so, or can define time based windows to use the timestamps instead of the system time? cheers Martin