Re: Question regarding parallelism

Hey Jerry, On Wed, Oct 21, 2015 at 11:11 PM, Jerry Peng wrote: > > When I submit the job, the number of task slots that gets used > (displayed on the UI) is only 20. Why is that? The total number of > tasks listed on the ui is 55. Do you mean the number of task slots is 55 (you just wrote task

Question regarding parallelism

Hello, I have flink streaming job as follows DataStream messageStream = env .addSource(new FlinkKafkaConsumer082( flinkParams.getRequired("topic"), new SimpleStringSchema(), flinkParams.getProperties())).setParallelism(5); DataStream> messa

Re: Zeppelin Integration

Hey Till, I cloned your branch of Zeplin and while it will compile, it fails tests on timeout, which consequently was the same issue I was having when trying to use Zeppelin. Ideas? --- Test set: org.apache.zeppelin.flin

Re: Session Based Windows

Hi Paul, good to hear that the windowing works for you. With the key based state I’m afraid you found a bug. The problem is that the state backend is not properly set to the right key when the window is evaluated. I will look into fixing this ASAP before the 0.10 release. Cheers, Aljoscha > On

Re: Session Based Windows

Hi Aljoscha, Thanks a lot for your Trigger implementation, definitely helped provide some direction. It appears to be working well for our use case. One thing I have noticed now that I have pulled the state API changes in is that key based state within a window function does not appear to be wor

Re: Zeppelin Integration

Hi Trevor, in order to use Zeppelin with a different Flink version in local mode, meaning that Zeppelin starts a LocalFlinkMiniCluster when executing your jobs, you have to build Zeppelin and change the flink.version property in the zeppelin/flink/pom.xml file to the version you want to use. If y

Re: Flink+avro integration

This is actually not a bug, or a POJO or Avro problem. It is simply a limitation in the functionality, as the exception message says: "Specifying fields by name is only supported on Case Classes (for now)." Try this with a regular reduce function that selects the max and it should work fine... Gr

Re: Flink Data Stream Union

I think the most crucial question is still whether you are running 0.9.1 or 0.10-SNAPSHOT, because the 0.9.1 union has known issues... If you are running 0.9.1 there is not much you can do except upgrade the version ;-) On Wed, Oct 21, 2015 at 5:19 PM, Aljoscha Krettek wrote: > Hi, > first of al

Re: Flink Data Stream Union

Hi, first of all, am I correct to assume that new SocketSource(hostName1, port, '\n', -1) should be new SocketTextStreamFunction(hostName1, port1, '\n', -1) or are you using a custom built SocketSource for this? If I replace it by SocketTextStreamFunction and execute it the example runs and prin

Zeppelin Integration

I'm setting up some Flink/Spark/Zeppelin at work. Spark+Zeppelin seems to be relatively well supported and configurable but the Flink is not so much. I want Zeppelin to run against my 0.10 build instead of the 0.6 build that ships with Zeppelin. My best guess at the moment on how to accomplish t

Re: Flink Data Stream Union

Hmm, that is strange. Could you maybe send the complete program so that I could have a look? > On 21 Oct 2015, at 16:43, Gayu wrote: > > No, it doesn't i even tried removing the filter and return all the values as > is received from the port. > > My doubt is, is there anything system or CPU sp

Re: Flink Data Stream Union

No, it doesn't i even tried removing the filter and return all the values as is received from the port. My doubt is, is there anything system or CPU specific that fails to attach the data to the datastream created. On Wed, Oct 21, 2015 at 10:39 AM, Aljoscha Krettek < aljoscha.kret...@gmail.com> w

Re: Flink Data Stream Union

So does the filter maybe filter out everything? > On 21 Oct 2015, at 16:18, Gayu wrote: > > Yes, exactly. > > On Wed, Oct 21, 2015 at 10:17 AM, Aljoscha Krettek > wrote: > So it is received in the filter but the print afterwards does not print? > > On 21 Oct 2015, at 15:49, Gayu wrote: > > >

Re: Flink Data Stream Union

Yes, exactly. On Wed, Oct 21, 2015 at 10:17 AM, Aljoscha Krettek < aljoscha.kret...@gmail.com> wrote: > So it is received in the filter but the print afterwards does not print? > > On 21 Oct 2015, at 15:49, Gayu wrote: > > > > The data does arrive in the second port and i am able to see that in

Re: Flink Data Stream Union

So it is received in the filter but the print afterwards does not print? > On 21 Oct 2015, at 15:49, Gayu wrote: > > The data does arrive in the second port and i am able to see that in the > filter class received. > It happens only on specific machine on which i run the code. > > > Yes, i did

Re: Flink+avro integration

Till Rohrmann wrote > What was your problem with using Java POJOs with the Scala API? Here's a quick example that demonstrates some of the problems I'm having. I used `max` in the example, but actually I get an exception for most of

Re: Flink Data Stream Union

The data does arrive in the second port and i am able to see that in the filter class received. It happens only on specific machine on which i run the code. Yes, i did forget to post here, but my program calls the unionMessageStreams() On Wed, Oct 21, 2015 at 9:39 AM, Aljoscha Krettek < aljoscha

Re: Flink Data Stream Union

Hi Gayu, could it be that no data ever arrives on the second input stream? Or that the filter filters out all messages? Also, in the example you posted you forgot to call unionMessageStreams(). Cheers, Aljoscha > On 21 Oct 2015, at 15:29, Till Rohrmann wrote: > > Can it be that you forgot to

Re: Flink Data Stream Union

Can it be that you forgot to call unionMessageStreams in your main method? Cheers, Till ​ On Wed, Oct 21, 2015 at 3:02 PM, flinkuser wrote: > Here is the strange behavior. > > Below code works in one box but not in the other. I had it working in my > laptop the whole of yesterday, but strangely

Re: Flink Data Stream Union

Here is the strange behavior. Below code works in one box but not in the other. I had it working in my laptop the whole of yesterday, but strangely today it doesnt work in my desktop. Can anyone please let me know what the issue is. public static void main(String[] args) throws Exception {

Re: Flink io files

Hey! These files are the spilled data from a sort, a hash table, or a cache, when memory runs short. If you have some very big files and some 0 sized, I would guess you are running a Hash Join, and have heavy skew in the distribution of the keys. Greetings, Stephan On Wed, Oct 21, 2015 at 12:5

Flink io files

Hi to all, what information can I infer from the files contained in the flink-io-xxx dir? what does represent each channel file and what does it mean when there are a lot of 0-size files and some very big files..? Best, Flavio

Re: Flink+avro integration

@Andrew Flink should work with Scala classes that follow the POJO style (public fields), so you should be able to use the Java Avro Library just like that. If that does not work in your case, please file a bug report! On Wed, Oct 21, 2015 at 9:41 AM, Till Rohrmann wrote: > What was your proble

Re: Cannot get Python API to work at all; java.io.FileNotFoundException for Flink python resources

Hi Ronkainen, Sorry for the late reply. Unfortunately, this is a bug in the Python API. I've reproduced the issue and fixed it for the upcoming releases. The fix will be included in the 0.9.2 and the 0.10 release. If you don't mind, you could already use the 0.10-SNAPSHOT version (0.10 release wil

Re: Flink+avro integration

What was your problem with using Java POJOs with the Scala API? According to https://issues.apache.org/jira/browse/AVRO-1105, the progress on adding a Scala API to Avro is kind of stalling. Cheers, Till On Tue, Oct 20, 2015 at 9:06 PM, aawhitaker wrote: > One more follow up: > > There doesn't a