Re: separation of JVMs for different applications

2016-12-06 Thread Manu Zhang
Good to know that. Is it the "standalone setup v2.0" section ? The wiki page has no Google-Doc-like change histories. Any jiras opened for that ? Not sure that will be noticed given FLIP-6 is almost finished. Thanks, Manu On Tue, Dec 6, 2016 at 11:55 PM Stephan Ewen wrote: > Hi! > > We are cur

Re: Thread 'SortMerger spilling thread' terminated due to an exception: No space left on device

2016-12-06 Thread Fabian Hueske
Hi Miguel, estimating the space requirements is not trivial. It depends of course on the algorithm and the data itself. I'm not an expert for graph algorithms and don't know your datasets. But have you tried to run the algorithm in a non dockerized environment? That might help to figure out if th

Re: Variable Tuple Type

2016-12-06 Thread Fabian Hueske
Hi Max, Tuples in Flink are of fixed length. You can define your own data types and serializers, but this is not the easiest solution. I would go for Array types, especially if your data can be primitive types (long). The serializer for primitive arrays should be almost as efficient as the Tuple

Quick Flink use case question...

2016-12-06 Thread Joe Olson
Suppose my goal is to answer the question: "Show me all the continuous time intervals a stock price was over X between times T1 and T2". The response I am looking for is an array of Tuple2s, each of which represent a start / end timestamp when the stock was above price X. There might be a lot o

Re: separation of JVMs for different applications

2016-12-06 Thread Stephan Ewen
Hi! We are currently changing the resource and process model quite a bit: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 As part of that, I think it makes sense to introduce something like that. What you can do today is to set TaskManagers to use one slot only, and then

Re: Should I decrease the taskmanager.memory.fraction ?

2016-12-06 Thread Stephan Ewen
Streaming applications currently do not benefit from this, which will hopefully change soon. Until then, you need not do anything, because the memory is lazily allocated, and will never be allocated for streaming jobs. On Tue, Dec 6, 2016 at 8:59 AM, Renkai wrote: > I've just read the article

Re: Flink Kafka producer with a topic per message

2016-12-06 Thread Stephan Ewen
You are right, it does not exist, and it would be a nice addition. Can you sketch some details on how to do that? - Will it be a new type of producer? If yes, can as much as possible of the code be shared between the current and the new producer? - Will it only be part of the Flink Kafka 0.10

Re: Default restart behavior with checkpointing

2016-12-06 Thread Maximilian Michels
Very good question! As the documentation mentions, the old way was to use `setNumberOfExecutionRerties` but it has been replaced by `setRestartStrategy`. If you don't configure anything, then your job will _not_ be restarted. However, if you have enabled checkpointing, then your application will b

Flink Kafka producer with a topic per message

2016-12-06 Thread Sanne de Roever
Hi, Kafka producer clients for 0.10 allow the following syntax: producer.send(new ProducerRecord("my-topic", Integer.toString(i), Integer.toString(i))); The gist is that one producer can send messages to different topics; it is useful for event routing ao. It makes the creation generic endpoints

Re: [DISCUSS] "Who's hiring on Flink" monthly thread in the mailing lists?

2016-12-06 Thread Kostas Tzoumas
yes, of course! On Tue, Dec 6, 2016 at 12:54 PM, Márton Balassi wrote: > +1. It keeps it both organized and to a reasonable minimum overhead. > > Would you volunteer for starting the mail thread each month then, Kostas? > > Best, > > Marton > > On Tue, Dec 6, 2016 at 6:42 AM, Kostas Tzoumas > w

Re: [DISCUSS] "Who's hiring on Flink" monthly thread in the mailing lists?

2016-12-06 Thread Márton Balassi
+1. It keeps it both organized and to a reasonable minimum overhead. Would you volunteer for starting the mail thread each month then, Kostas? Best, Marton On Tue, Dec 6, 2016 at 6:42 AM, Kostas Tzoumas wrote: > Hi folks, > > I'd like to see how the community feels about a monthly "Who is hir

[DISCUSS] "Who's hiring on Flink" monthly thread in the mailing lists?

2016-12-06 Thread Kostas Tzoumas
Hi folks, I'd like to see how the community feels about a monthly "Who is hiring on Flink" email thread on the dev@ and user@ mailing lists where folks can post job positions related to Flink. I personally think that posting individual job offerings in the mailing list is off-topic (hence I have

Re: microsecond resolution

2016-12-06 Thread Niels Basjes
Hi, If you set your stream to use event time (i.e. env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); ) I think you can use all functions in the system without modification. So there is no need for any custom window assigners. If you then put microseconds into the timestamp field the y

Should I decrease the taskmanager.memory.fraction ?

2016-12-06 Thread Renkai
I've just read the article https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html, It seems that the MemorySegment mainly used for sort and join for batch execute. But I mainly use Flink for streaming work, I do a lot of map/flatMap and reduce in Flink. Is the MemorySegment stil